What might be the impact of allowing learners influence the design of e-learning tools? The need for an integrated response. Dr. Tom Farrelly, Sarah O’Toole and Tony Murphy (Institute of Technology, Tralee)
Contents Introduction ................................................................................................................................ 1 Previous efforts to gather feedback............................................................................................ 2 The Project ................................................................................................................................. 2 The Cohort Profile .................................................................................................................. 2 The Workbooks ...................................................................................................................... 3 The Survey ............................................................................................................................. 5 Why real-time feedback? ........................................................................................................... 5 Survey Responses ...................................................................................................................... 6 Issues & Responses ................................................................................................................ 7 Discussion ................................................................................................................................ 11 Constructing a new team ...................................................................................................... 12 Redefined Roles ................................................................................................................ 13 Conclusion ............................................................................................................................... 15
Introduction Through the use of real-time feedback, learners were offered the opportunity to have an impact on the development process of three online workbooks used to support face-to-face teaching of an Introduction to Social Research Methods module in a blended learning environment. As might be expected, real-time feedback (through the use of an online survey tool) influenced the nature and manner of the response to technical troubleshooting and altered the way in which content was constructed and presented. However, an unexpected outcome of the process was the manner in which real-time feedback impacted on the roles of support staff and the process behind the construction of these tools. It redefined the 1
relationship between the support staff and led to the development of a multi-skilled e-learning support team that now sees how important it is to bring the learner in to the design process as early as possible. In this chapter, we examine the impact of incorporating real-time student feedback into the development process of three online workbooks and the resultant development of an integrated e-learning development team in response to this feedback. The details of the survey and responses are then considered, as is the rationale behind using real-time feedback. The impact of that feedback is then examined in terms of how it affected troubleshooting, the process of developing individual reusable learning objects (RLOs) and complete online workbooks and the roles of those involved in that process, including the learner.
Previous efforts to gather feedback Prior to the design of the three workbooks, online surveys using SurveyMonkey™ had been used by two of the team to assess students’ experience of ebrary books (an online book provider) as part of a study that looked at the impact of targeted library training on the use of this online resource. Participation in this project brought home the value of student feedback in designing services and presenting resources to students.
However, as the survey was a
retrospective investigation, undertaken after the module’s completion, therefore lessons learnt were of no immediate benefit to that particular student group’s learning experiences. Where the study was of value to the students surveyed was in the manner in which it opened a line of real-time communication that had not previously existed. Participants involved in the study were informally feeding back on a number of issues, not all related to the study. The value of this line of communication subsequently influenced decisions about how to capture feedback for the online workbooks for the Social Research Module.
The Project This section provides a brief outline of the project in terms of an outline of the groups involved, an exposition of the online workbooks and a brief description of the survey methodology. The Cohort Profile The target population was drawn from three undergraduate first year student groups: Social Care (n= 64), Early Childhood Care & Education (n= 39) and Youth & Community 2
Work (n = 30) thus totalling 133 students who were all undertaking a first year introduction to social research module. Given the relatively large class sizes and the limited opportunity for tutorial support, it was decided to incorporate a strong e-learning element into the module’s teaching and learning strategy that was intended to be used by students in an independent manner as an on-going resource to support the face-to-face delivery. While these students had been introduced to accessing their notes online through the VLE Backboard, they had not used learning resources as their main source of learning material before. The groups were provided with the Blackboard training which covered the basics, such as logging in, the different menu options and opening content. They also received the standard library induction consisted of a guide to searching the library’s online catalogue and navigating the library web site, a brief introduction to searching online databases of journal articles of ebooks and a brief reference to using such search techniques as Boolean operators, truncation and phrase searching. They were also provided with a brief introduction to navigating workbook one and the various components. However, there was no attempt to provide any ‘specialised’ training as this was primarily intended to be used as an aid to teaching and learning and not a research exercise per se. The Workbooks The teaching strategy took the form of a combination of face-to-face lectures and the development and use of three online interactive ‘workbooks’ that were published within an Articulate (www.articulate.com) platform. Each ‘page’ of the workbook incorporated some form of online teaching element such as: an embedded video, hyperlinks to ebooks and/or a websites, directed reflective activities, assessments, interactive quizzes, reading material and PowerPoint presentation videos, screencasts using Camtasia Studio (software for screen recording and video editing - www.techsmith.com) were also incorporated into the workbooks.
3
Figure 2 - Workbook One Cover
Figure 1 - Self-Assessment Feedback
Figure 5 - Embedded Videos
Figure 4 - Hyperlink to Electronic Book
Figure 3 – ‘Flip’ Book
4
The Survey As soon as each of the three workbooks went live a hyperlink to each survey was made available via Blackboard in the research methods module and via an email sent to all class members. Students were asked to undertake the survey as soon as they had completed a workbook. The survey was created in SurveyMonkey™ and consisted of five questions; four closed questions which focused on the technical aspects of using the resources, whether they were having network issues, could they access each section of the resource and did the tools open up and run proficiently and a fifth open-ended question which allowed for free comments.
Why real-time feedback? The decision to undertake real-time feedback as part of the development process was driven by local considerations and emerging best practice as reported in the literature. Prior to this study, there had been little opportunity to capture and evaluate feedback as part of the design process with RLOs.
The environment that the Instructional Designer had been
working in did not facilitate feedback because RLOs tended to be stand alone and did not come under the same review scrutiny as face-to-face learning. Seen as complementary and not intrinsic to the learning experience, RLOs were not part of a “course board” review and therefore the need to collect user feedback was not so great. With this project, the move from creating a one-off learning object (which acted as an additional learning resource) to the development of a blended module, where the main sections of content were made up of a series of learning objects (the workbooks), highlighted the importance of implementing a design process, which would also include user feedback. Fisher and Miller (2008) note that the early collection of data allows educators to respond to unique concerns so that potential problems are not allowed so develop. Similarly, Lukus (2007) collected responses from students on a lecture-by-lecture basis and found that lecturers received more timely and specific feedback on their performance compared to endof-course evaluations. In recognising the value of continuous improvement, Fisher and Miller (2008) also argue that evaluation is an overall process that starts with the first formal interaction between students and educators thereby creating a holistic rather than a disconnected one-off approach.
5
The perceived expectations of the learners were also a consideration in opting for the early collection of data. New Learners, also known as digital natives or Generation Z, would expect to be able to give and receive instant feedback.
According to Doiron and Asselin
(2011) these students are “action-oriented problem solvers and see technology as their primary tool for engaging with the world… [which they use]… to communicate with known and unknown others and to shape their social and work lives” (p.225). Taking an ‘act first and see what happens approach’, the new learners’ “natural learning mode seems to be that of multi-tasking, multi-modal and multi-resourcing” (Doiron and Asselin, 2011, p.225). Some would argue that student demands as they exist today and the integration of technology into society make online courses inevitable (Brooks, 2009), which would make sense in that the online learning environment appears to be necessary to accommodate such multi-faceted needs. A number of studies, (Enriquez (2010) and Beuckman, Rebello & Zollman (2007), have looked at using technology within the classroom or lecture theatre to facilitate immediate feedback or instant polling in an effort to improve student performance. These studies are based on the accepted logic of educational research that students learn best when they actively construct their own knowledge (Beuckman, Rebello and Zollman, 2007). The value of real-time feedback is also acknowledged in the business world. ‘Saba’, a developer of learning management tools for business, recently added Impressions, which is a tool that allows for informal instant performance feedback between mentors and mentees.
Survey Responses There were 133 students between the three courses that were asked to complete the survey; therefore over the three surveys the potential population was 399. There were a total of 161 responses from the three surveys which represented an average response rate of 40% as can be seen from the table below. However, it is important to note that while the overall response rate remained quite constant, the constituent groups and individuals necessarily varied.
6
Survey #
Social
Early
Youth &
Care
Childhood
Response
Community Rate
Survey 1
34
13
8
55 (41%)
Survey 2
23
19
9
51 (38%)
Survey 3
29
19
7
55 (41%)
Given that the response rates for all three surveys remained consistent at approximately 40% of the target population, it is reasonable to conclude that a substantial minority of the students saw participation in the process as being of some benefit and meaning. Issues Although our intention was to utilise the multi-media capabilities as much as possible, we were very aware that if there were difficulties in accessing the various elements their use as an educational resource would be greatly diminished. For ease of presentation the reported difficulties is presented in one table. However, as the respondents were by virtue of selfselection not necessarily the same people each time, the data should read (as we did) as selfcontained responses to each workbook. The key issue is that we had the opportunity to respond to the difficulties as they arose and were reported rather than wait until the module was finished. This issue is particularly evident in the team’s response to the Ebrary book links problem as noted by the ‘spike’ of five people who could not access the links in workbook two as illustrated below. Could
Lot
of Some
Not
Difficulty Difficulty Little
Access ‘Flip’
Very
No Difficulty
Difficulty
4
4
11
9
22
2
1
16
11
20
2
4
11
10
21
0
0
10
11
26
Books
Workbook One
Videos
W
YouTube
‘Flip’
Ebrary Book
o
w
T
k
o
o
b
k
r
o
Links
7
Books YouTube
0
2
8
10
24
5
2
8
13
16
Embedded 4
4
12
17
15
1
9
17
22
Videos Ebrary Book
Workbook Three
Links
Videos Ebrary
2
Book Links
Six weeks into the module’s commencement, ebrary’s homepage changed to offer subject searches from images. The image links worked on-campus because on site access is verified by Internet Protocol (IP) address. Off-site access is redirected through a proxy server for authentication. The redirect is built into the Uniform Resource Locator (URL) which is unique for each member institution that uses the same authentication method. The image subject searches did not have the correct URL for off-site access, and the first users to notice this failing were participants in the study, over a weekend. The problem was instantly fed back to the systems librarian (Tony) through the lecturer. A temporary workaround was devised with a screenshot guide, which the lecturer (Tom) sent out to the students. On the following Monday, the systems librarian was able log a call with Ebrary before altering the landing page for access that only contained links that worked both on and off site. While it was important to be responsive and amend the material where appropriate; it was also important to recognise our limitations in terms of the range of issues that we could incorporate into our design and response process. There were after all a number of issues that were not within our remit or more importantly: our control such as Internet connection speeds. For example, one of the principal technical difficulties consistently reported by many studies into E-learning is connectivity issues, notably access to high speed quality broadband. Therefore, one of the questions we asked was how the respondents accessed the workbooksvia ‘Dial-Up, Broadband (Wireless or Wired) or a Dongle? By cross-tabulating the responses
8
between degrees of difficulty reported in accessing some of the features (‘Flip-Books’, Videos and Ebrary Books Online) and connectivity modes we were able to discern if connectivity rather than inherent design/technical features could be the problem. While we knew that we could not cover for all eventualities, it was important that if there was something we could fix e.g., a web link not working or a video file not opening then we needed to incorporate that feedback from the students as soon as possible to ensure students could view the workbooks. In terms of connectivity, dial-up was not surprisingly the least used connection method; ranging from six respondents in survey one dropping to three by survey three with the rest connecting via broadband through a variety of methods. While the people who used dial-up did experience greater difficulties than the broadband users, some broadband users also encountered a number of difficulties which we were able to address. Across the three surveys an average of 70.7% of students accessed the workbooks from home, while only 9.8% of students accessed the workbooks from the college labs (where the network would be the fastest) and 16.5% using the library. Again, the average figures across the three surveys indicated that 81.7% of students accessed the resources from their laptops with an average of 17.23% using a mobile device or smartphone. Seeing these figures highlighted how important it was that students have access to files that are not too large as they are not on the college network and that students are using mobile devices to view material. When one examines the responses to the open-ended questions, it is clear that there were a number of discernible themes to emerge over the ten week period in which we developed and published the three workbooks. While we were of course keen ascertain if there was any issues or problems with the workbooks, we were equally keen to assess the value (if any) placed on the workbooks by the students. There were a number of issues and problems that were reported. Nonetheless, it was evident that once some of the early issues in workbook one were attended to, the majority of respondents had had positive experiences and felt that using the workbooks was a useful learning exercise as exemplified by the following selection of comments:
“Great learning activity” (Workbook 1)
“I found it very easy to use and had no problems at all” (Workbook 1) 9
“Very easy to understand and easy to access” (Workbook 2)
“This is one of the best ways of learning I have ever experienced. I love it.”
(Workbook 2)
“Made the learning easier” (Workbook 3)
While these positive comments were welcome, the positive comments that referred to specific elements and activities proved to be particularly informative:
“Great to be able to go back on the video's to understand where the x and y
figures go as this can be confusing” (Workbook 3)
“Loved the quiz it really made me think about the way in which things are
organised”. (Workbook 2)
“The book seemed to really simplify things with great clarity the quiz is great
to see if you have grasped an understanding of the topic” (Workbook 2)
“I found the videos a lot more interesting than just reading a book, the only
drawback was that the videos could not be paused” (Workbook 1) The last point is a particularly good example of how well the real-time feedback and response element of this process worked. In workbook 1 there were a number of respondents who noted that while they found the videos useful, they wished that they could pause the embedded YouTube videos:
“It would have been much more helpful if a person could pause when they
need to take down a note, instead of having to start at the very beginning every time”.
“The only inconvenience was not being able to pause mid-slide when
interrupted” As the survey was monitored on a daily basis this issue quickly came to light and the team were able to deal with this straight away. One example of this was with a set of five short videos that were part of a workbook. The videos originally were added to the workbook without a control bar. Students advised that it would be more beneficial if they could pause and stop the videos when they wanted. We were able to republish this quickly and add a control bar to each video. The previously discussed ebrary link problem was also noted in the open-ended responses. In fact it was the reference to the Tralee area that provided a useful starting point with which 10
to help discern the nature of the problem quickly indicating that the problem seemed to be an off-site issue as indicated from the quotes in response to workbook two:.
“Difficulty at home ebrary book links”
“Couldn’t get sent through to the ebrary, but do like the idea of being directed
to a specific text as you are going through the workbook”
“I could not access the ebrary. It said I needed to be in the Tralee area”
As previously argued, if the survey response rates are taken as an indication, students appeared to feel that it was worthwhile to complete the surveys as they felt their feedback would be read and incorporated into future iterations. Students realised that they would see their changes implemented and understood that their contribution was appreciated as highlighted by these three responses:
“The only problem experienced at any stage was accessing the first book on
stats from home which was rectified and gaining access was no bother” (Workbook 3)
“The fact that I could now pause the videos was very good” (Workbook 1)
“Had a problem the first night on accessing Ebrary book links but this seemed
to have been rectified the second night” (Workbook 2) While the feedback in real-time was highly informative and certainly improved the development of the workbooks; on reflection it was the simultaneous development of a different way of working that was the most important impact of the real-time feedback as discussed below.
Discussion Sims (2006) asked “how valuable might strategies be where learners are the proactive providers of content, not just recipients?” (p.2). Incorporating real-time feedback from workbook one into the construction of subsequent workbooks appears to have allowed the learners to become more like proactive providers of content. The real-time feedback did more than simply help troubleshoot; rather it provided the student with a place at the table during the development stages. In an ideal world, all new e-learning material would be subject to a rigorous design and development process with attendant usability testing. In reality, many of those who we would like to see develop and use these resources (the lecturers) are increasingly time poor, given 11
the increase in their roles and responsibilities. If we are going to encourage more people to engage in the development of online learning objects we need to have an easier more flexible and more responsive process. Drawing on our experiences; the incorporation of real-time feedback combined with an integrated team helped speed up the development process as we could design, deliver and amend in an almost contemporaneous manner. Utilisation of real-time feedback in conjunction with an integrated real time response team should help speed up the design process and thus encourage more lecturers to engage in RLO Development. While the realtime feedback was invaluable we would contend that it was only half the equation; there is little point in having the responses from students without having the ability to produce and integrated quick response. Constructing a new team As the process of creating online tools has developed, there has been a concomitant change in the nature of the relationship between the different elements (Kanuka, 2006; Corrall and Keates, 2011).
Locally, this project led to the development of a team with
redefined roles for each member. Working together, the lecturer, the instructional designer, the systems librarian and the students through the real-time feedback each could analyse the resources from different viewpoints so a more in depth evaluation of the resources was carried out. The four roles also provided a technical and pedagogical support to one another as part to the design process, the effect of the combined roles working together was greater than one individual working alone. The value of a team-based approach to web-based learning has been acknowledged elsewhere. Laurillard (2002) says that “collaborative development is crucial for learning technologies because of the range of skills needed.” (p. 227) Similarly, the OU Supported Open Learning model, which Fleck (2012) argues sets the standard for quality delivery at scale, consists “of dedicated faculty focused on course design, course team produced materials, structured support from associate lecturers (tutors) and professional logistics infrastructure for scale delivery” (Fleck, 2012, p. 401). Barajas and Gannaway’s (2007) study of online learning at the University of Barcelona concluded that six dimensions—institutional strategies, intellectual property rights, library services, teacher training and support, student support, and virtualisation of content and course design—are integral to any program of e-learning integration. While the authors 12
would argue that these elements can be treated separately, our project would indicate that taking an inter-connected approach to the integral elements behind e-learning might benefit the learning experience for students. Similarly, Russell and Koppi (2007) claim that there is a move away from “individual lecturers all working on separate e-learning courses, as early adopters…[towards]…organized groups who are able to set up shared systems and use support staff instead of the lecturers doing everything themselves” (p.11). Closer to home, MacKeogh and Fox (2008) claimed that their study of the integration of e-learning on the Dublin City University campus revealed that “e-learning is not just the responsibility of academics; administrative support units are key facilitators, including the Learning Innovation Unit, Library, Student Services, Computer Services, Registry, Finance, Human Resources etc.” and that for “e-learning to flourish, all systems must interact to ensure that there are no blockages or inhibitors” (p.150). Redefined Roles For most instructional designers who work in an academic institution, a large percentage of their work involves training academic staffs on the different applications and tools that are available. They also work with a lecturer on the development of a resource, which once complete is given to the lecturer and the instructional designer has no more to do with it. They do not get the chance to engage with the end user of the resource (the student). Getting the real-time feedback from the students and then incorporating it into the workbooks allowed the instructional designer to become more aware of the end users’ needs and of the environment of the lecturer where requests and unscheduled changes need to be worked on. This level of contact with the students provides a more engaging environment for the development of RLOs, and while this may increase workloads or increase the demand to changes to be made, this is balanced out by working with the systems librarian to incorporate changes and improve resource usage. The change in role for both the systems librarian and indeed libraries brought on by this project was twofold. Firstly, there was the adoption of a new role: the embedded librarian. The idea of an embedded librarian dates back to the early 1990s (Tyler and Shumaker, 2007) and the 1993 article from Tom Davenport and Larry Prusak, “Blow Up the Corporate Library,” which encouraged librarians get out of the library and embed themselves in an organisation’s information needs. A sentiment echoed by the OCLC’s E-Learning Taskforce in 2003 and reiterated by Gibbons (2005, p. 12) where she argues that: “to remain relevant, 13
academic libraries must go where the students and faculty are.� More to the point, libraries need to be where the learning is happening, even if this is the virtual environment of a CMS. Secondly we come to the concept of the blended librarian which originated in 2004 with Steven Bell and John Frank, who sought to encourage academic librarians to integrate instructional design and technology skills into their existing library and information technology skill set. Participation in this study, and the direction that the real-time feedback led the support staff, meant that the systems librarian became more involved in helping to develop the course content, thereby becoming more embedded. It also required the systems librarian to up skill in the area of technology to support e-learning. Essentially, the project suggests a redefinition of the role of the systems librarian to become an ‘emblended’ librarian. Finally, for the lecturer, if e-learning provision is to move from beyond the realm of early adopters to more mainstream adoption, colleges need to foster and develop more innovative and proactive methods of the development. Lecturers trying to incorporate e-learning into their teaching and learning strategy, but largely working alone, run the risk of burnout and simply giving up. They also risk diluting the core value of what they have to offer the design process by trying to do everything.
What this project is suggesting is that e-learning
provision requires team, multi-skilled approach involving other staff and the learner, which for the early adopter lecturers may mean stepping back and having less roles than previously.
14
Conclusion
Gathering feedback is only worthwhile if you have the ability to respond and incorporate the findings. Our ability to respond quickly was due to a commitment to a strong sense of collegiality and teamwork with the three elements of the design team working closely: the systems librarian, a subject lecturer and the educational technologist. Ultimately, the learning environment was improved because the real-time feedback allowed for student needs to be incorporated into the construction of content and its delivery, a quick troubleshooting response and the creation of a multi-skilled support team. Sims (2006 p. 2) argues that if we are to adopt a constructivist and collaborative online environment we need to engage in a meaningful a participatory way with learners, particularly the new generation of learners, “for whom technology IS the environment and for who learning means different things”. He goes on to argue that “If we adopt this assumption, then we must re-think the paradigms for conceptualising, creating and implementing online learning environments”. Thus the use of real-time feedback and response became more than an exercise in information sharing; it became an attempt to meaningfully include e-learners in the creation of their online environment. In this chapter we were mindful of not simply presenting the results of a survey. Essentially, the survey feedback provided a jumping off point through which we were able to elucidate the lessons learnt from engaging in our process of developing online resources. If elearning is to move from being a peripheral to a mainstream activity for the vast majority of lecturers a more supportive model than is currently being offered is necessary. We hope that this chapter offers one such way of conceptualising how this might be achieved.
15