Short Paper Proc. of Int. Conf. on Advances in Computer Science and Application 2012
Measuring e-Learning Technology Erlan Burk Park University, Parkville, Missouri, USA Capacity for sufficient information and the incentive to change resulted in complexity. Complexity is the ability of individuals to have many roles. Three principle advances in information technology include random access storage, graphical displays, and clientserver architecture. Innovative features in these technological advances increased online classroom power in three ways. Capacity increased through increased interaction. Volatility increased through presentation of information. Complexity increased through availability of information (Real, 2006).
Abstract. - Measuring the success of technology learning in cyberspace includes; (a) evaluation with a model by Kirkpatrick, (b) key indicators with a model by Burk, and (c) organizational behavior using model by Dervitsiotis. In addition, a paradigm for measuring the capacity of technology supporting e-learning is proposed. Further research on this paradigm has the potential of extending the measurement from an e-learning environment to a general cyberspace organization.
I. INTRODUCTION Measuring the effect of technology support for e-learning can be accomplished (a) evaluation using a model by Kirkpatrick (2010), (b) key indicators using a model by Burk (2011), and (c) an organizational behavior model by Dervitsiotis (2008). Evaluation using the model by Kilpatrick and Burk provide implicit measurements for supporting technology and other factors, such as the design of the elearning process. However, the model by Dervitsiotis offers the potential to measure components of technology support explicitly. Further research on this paradigm has the potential of extending the measurement from an e-learning environment to a general cyberspace organization Over 50 years ago, Kirkpatrick (2010) introduced an approach for training evaluation using a four level model. Level one of the model evaluated training by measuring the perceived reaction of trainees to the training experience. Level two of the model evaluated participant learning by measuring the perception of knowledge and skills for participants resulting from the training. Level three of the model evaluated participant behavior by measuring the perception of transfer for skills and behavior to the workplace. Level four of the model evaluated the training by measuring perception of the results and benefit of the training to the organization. The online classroom is a virtual meeting place where students spend hours each week in study. Proper classroom design makes the classroom environment inviting, organized, functional, friendly, and comfortable. Improper classroom design can cause an unsuccessful learning experience for students. However, design issues in a classroom cause the learning experience of students not to be optimal. With none of the issues, indications are that the classroom is an environment for student success. (Burk, 2011) Dervitsiotis (2008) found that organizational success depends on members of the organization having capacity, volatility, and complexity. Successful interaction of members in an organization resulted in the interconnectivity and interdependence. Interconnectivity provided the capacity for the organization to contain sufficient information to reach organizational objectives. Interdependence prompted volatility in the organization and provided incentive for change. Š 2012 ACEEE DOI: 02.CSA.2012.01. 4
II. EVALUATION OF E-LEARNING Sweetland (1996), van der Merwe (2010), along with Barcala, Sanzo Perez, Jose, Trespalacios, and Juan (1999) used the Kirkpatrick model to study human capital theory. Findings were that educational investment in people has benefits to individuals, as well as in society. In addition, they found business performance improved through education and training. Consequently, the results of research verify the model described by Kirkpatrick (2010) is appropriate for evaluation of training in an online class. Phillips and Phillips (2008) expanded the Kirkpatrick model by adding a fifth level. This level calculated the return on investment for a business. However, Sagafian (2011), in his review of the literature, found issues with use of the fifth level for training evaluation proposed by Phillips & Phillips. Measuring return on investment with the fifth level targeted only one organization since measurement standards change between companies. In the global business environment, the norm of a business operation is more than one company. Consequently, the basic Kirkpatrick model, without the expansion by Phillips & Phillips has limitations. An example of successful use of the Kirpatrick model was the study by Vellios and Kirkpatrick (2008). They demonstrated the usefulness of the model in their evaluation of management training for a healthcare organization in New York. The collection of data used an interview and a survey containing the four levels of the Kirkpatrick training evaluation model. The analysis of level one data showed an eagerness of trainees to use their training. Analysis of level two data showed a distinct learning experience for the trainees. However, analysis of the level three data revealed a gap between the training experience and use of training in a working environment. There were numerous requests from organizational management for retraining on poor-performing employee management, resolution of employee conflict, and communication ability. Past training had not been effective in resolving these issues. Organizational statistics revealed a 22% increase in complaints regarding management. 93
Short Paper Proc. of Int. Conf. on Advances in Computer Science and Application 2012 Results of the study by Vellios and Kirkpatrick (2008) prompted the organization to incorporate corrective actions steps in the training function. In addition, work goals now included appropriate training. These changes caused training culture to develop within the organization. The study by Villios & Kirkpatrick also prompted the training of experts to conduct training evaluation surveys using level three of the Kirkpatrick model. The scenario studied by Vellios and Kirkpatrick (2008) for management training could just as well be a scenario for technology training. Although limitations exist for use of the Kirkpatrick (2010) model, this model is widely used for evaluation of training in cyberspace. Potential use of this approach for online technology training can be effective and is suggested.
as possible. Also, assignments should be modular or ‘stand alone’. Geography of students assigned to groups can cause difficulty in communication. Communication between members of the group becomes easier when the geographical area of the group is smaller. Consequently, a recommended approach for group composition is directing students to form groups based on their geographical location. Exams without verification of identity of the student may cause academic standard issues. To assure the valid measurement of student learning, many institutions employ proctored testing. The student shows an identification containing a picture to the proctor, then completes the exam under proctor supervision. The proctor then returns the completed final exam to the instructor. Proper course design results in an enhanced learning experience. When design issues are absent from the classroom, a course is inviting and comfortable. With none of the issues, indications are that the classroom is an environment for student success.
III. KEY E-LEARNING INDICATORS The online classroom is a virtual meeting place where students spend hours each week in study. Proper classroom design makes the classroom environment inviting, organized, functional, friendly, and comfortable. Improper classroom design can cause an unsuccessful learning experience for students. A paper by Burk (2011) outlines design issues that might occur in an online class. A review of a classroom, finding one or more of these issues, indicates the learning experience of students will not be optimal. More time can be spent learning course logistics than learning course topics. Moving through the classroom should be easy for the student. Understanding assignments can require more time than working on them. An assignment should include only the information necessary for completion. Not enough information causes additional searching for assignment details. Too much information creates confusion Incremental assignments can eliminate peak loads in the class workflow. The structure of assignments should be for completion of a topic, or subject, or learning objective, a piece at a time. This structure gives the flexibility for a particular part of the topic to move through the sequence of study. Unreasonable time limits detract from learning. Assignments during the week, and from week to week, should require about the same amount of time to complete. Should the load level prove to be different than estimated, the course design needs modification. Participation assignments are counter-productive to learning objectives. The timing of assignments during the week should be such that one is due each day of the week. Proper scheduling of assignments automatically provides the participation required by an institution. Non-individualized assignments provide opportunity for plagiarism. Using incremental and individualized assignments eliminate plagiarism. There is no opportunity for plagiarism since each student has a different assignment. Confusion results from multiple learning experiences at the same time. Assignments should be separated into sections with students interacting in the classroom as much Š 2012 ACEEE DOI: 02.CSA.2012.01. 4
IV. SUCCESS OF E-LEARNING ORGANIZATIONS Dimensions of an organizational environment are measure of organizational success. The success of the online organizational environment directly relates to the effect of online training. Dervitsiotis (2008) reviewed organizational environment dimensions. He performed analytical research of organizations finding results that apply to the study of online training. The study by Dervitsiotis (2008) reviewed advances in technology, organizational demographic changes, and world economy globalization with a focus on organizational survival. The review used graphical analysis to study organizational capacity, volatility, and complexity in selected global organizations. The analysis focus was on four stages of growth in evolution of organizational improvement. From this analysis, Dervitsiotis (2008) found that organizational success depends on members of the organization having capacity, volatility, and complexity. Successful interaction of members in an organization resulted in the interconnectivity and interdependence. Interconnectivity provided the capacity for the organization to contain sufficient information to reach organizational objectives. Interdependence prompted volatility in the organization and provided incentive for change. Capacity for sufficient information and the incentive to change resulted in complexity. Complexity is the ability of individuals to have many roles. Success in an online training course is consistent with the findings of the study by Dervitsitois (2008). Use of advanced information technology increased the capacity for successful interaction in the online training classroom. This reduced the time required for classroom activities. The reduction in time allowed time for new activities, providing the incentive for change and adding volatility to the organization. An increase in interaction, and the incentive to 94
Short Paper Proc. of Int. Conf. on Advances in Computer Science and Application 2012 server architecture. Innovative features in these technological advances increased online classroom power in three ways. Capacity increased through increased interaction. Volatility increased through presentation of information. Complexity increased through availability of information (Real, 2006). Invention of the magnetic disk introduced random access for storage and retrieval of data (Hoagland, 2005). The result was a significantly shorter time to process data. Increased speed of in processing data has enabled increased organizational capacity in the form of student-to-student and student-to-teacher interaction. This increase in interaction increased organizational power (Mavrou, 2010). Graphical display advances replaced one-line-at-a-time mechanical text devices to interface with users in an organization. For example, early data displays were on devices such as a typewriter or line printer (Kewney, 2008). Enhanced presentation of information increased volatility through increased choices in reading and processing data. This increase in volatility resulted in greater organizational power (Badler, 1999). Client-server architecture replaced the dumb-terminal interface between the computer and a member of an online class. The dumb-terminal displayed and collected data, but had no processing ability (Evans, 1980). Replacing the dumbterminal with a computer enabled processing of data jointly at the main computer and the user computer. This dual process increased availability of information and provided opportunity for more complex data in the online classroom. The increase in complexity of information increased organizational power (Domanski, 1994). Hill (2005) studied the increase in organizational from use of technology. In addition, Freidman (2008) explained increased organizational power from significant advances in information technology. Random access data storage, graphical displays, and client-server architecture were technologies having significant influence on organizations. An increase in organizational power for an online class results in an enhanced learning experience for students, and consequently, is a measure of success.
change, increased the ability of trainees to have many roles, and to handle the complexity of a greater number of tasks. The first measurement of organization success mentioned by Dervitsiotis (2008) was capacity. Yang, Yeh, and Wong (2010) studied the organizational capacity for interaction in a virtual learning community. The findings of Yang et al. regarding the relation of organizational capacity to organizational success were consistent with those of Dervitsiotis (2008). The second measure of organizational success described by Dervitsiotis (2008) was volatility. Volatility is ability to change. Charles, Lauras, and Van Wassenhove (2010) studied volatility in an organization similar to an online training classroom. They reviewed the literature, performed a case study using symbolic modeling. The initial study on volatility by Charles, Lauras, and Van Wassenhove (2010) focused on humanitarian organizations. In those organizations imbalance between supply, demand, and disruptions called for a high level of volatility. The definition and model for volatility developed for humanitarian organizations also applied to the study of supply and demand in profit making organizations. Charles et al. verified usefulness of the model by using it for assessment of supply chain volatility. The relation between volatility and organizational success described by Charles, Lauras, and Van Wassenhove (2010) for humanitarian and profit making organizations were consistent with volatility in an online training classroom. In the online training classroom, information presentation supplies information to trainees and meets their demands for completing classroom activities. The volatility resulting from this supply of information supports trainees learning skills and knowledge. The third measure of organizational success described by Dervitsiotis (2008) was complexity, or ability of individuals in an organization to have many roles. The study by Kimmerle, Moskaliuk, Cress, and Thiel (2011) was for complexity in an organization similar to an online training classroom. Kimmerle, Moskaliuk, Cress, and Thiel (2011) found that a pivotal task in all organizational activities was management of complexity when individuals interact. The aspects of complexity found by Kimmerle, Moskaliuk, Cress, and Thiel (2011) are consistent with the complexity in an online training classroom. Availability of information in the online training classroom increased complexity and interaction. Moreover, to realize this increase, classroom activities must have available information in the appropriate quantity, at the appropriate time for effective trainee interaction in online training. Otherwise, there is a decrease in the interaction of trainees in the online classroom. In summary, Dervitsiotis (2008) found that organizational success depends on members of the organization having capacity, volatility, and complexity. These dimensions are useful in evaluating online technology training.
VI. FURTHER RESEARCH Indications are that a direct relationship exists between technological advances and the success of an online classroom. The organizational dimensions of capacity, volatility, and complexity defined by Dervitsiotis (2008) each have a key relationship to data storage, graphical displays, and client-server architecture, respectively. Although there may be other technological factors affecting organizational dimensions, a study of these key factors have the potential to build a planning model for an online class, and subsequently for any organization in cyberspace.
V. CYBERSPACE TECHNOLOGY CAPACITY Three principle advances in information technology include random access storage, graphical displays, and clientŠ 2012 ACEEE DOI: 02.CSA.2012.01.4
95
Short Paper Proc. of Int. Conf. on Advances in Computer Science and Application 2012 REFERENCES
[11]. Kewney, G. (2008). Step back in time. Personal Computer World, 2(1), 10. [12]. Kimmerle, J., Moskaliuk, J., Ulrike, C., & Thiel, A. (2011). A systems theoretical approach to online knowledge building. AI & Society, Journal of Knowledge, Culture, and Communication, 26(1), 49. [13]. Kirkpatrick, D. (2010). The four levels are still relevant. T+D Magazine 64(9), 16. [14]. Mavrou, K. (2010). Researching computer-based collaborative learning. British Journal of Educational Technology, 41(3), 486. [15]. Phillips, J., & Phillips, P. (2008). Distinguishing ROI myths from reality. Performance Improvement, 47(6), 12-17. [16]. Real, J. (2006). Information technology as a determinant of organizational learning. Industrial Marketing Management, 35(4), 505-521. [17].Saghafian, M. (2011). A critical review of research on technology-based training in business organizations. Research in Higher Education Journal, 10, 1-10. [18]. Sweetland, S. (1996). Human capital theory: Foundations of a field of inquiry. Review of Education Research, 66.(3), 341. [19]. van der Merwe, A. (2010). Does human capital theory explain the value of higher education? A South African case study. American Journal of Business Education, 1(1), 107-118. [20]. Vellios, G., & Kirkpatrick, D. (2008). On the level. T+D Magazine, 62(12), 26-30. [21]. Yang, Y., Yeh, H., & Wong, W. (2010). The influence of social interaction on meaning construction in a virtual community. British Journal of Educational Technology, 41(2), 287.
[1]. Burk, E. (2011). Online learning indicators. Association for Computing Machinery e-Learn Magazine, 7, July 2011. [2]. Badler, N. (1999). Animation control for virtual humans. Communications of the Association for Computing Machinery, 42, 64. [3]. Barcala, M., Perez, P. , Jose, M., Gutierrez, T., Antonio, J. (1999). Training in small business retailing; testing human capital theory. Journal of European Industrial Training, 23(7), 335-352. [4]. Charles, A., Lauras, M., & Van Wassenhove, L. (2010). A model to define and assess the agility of supply chains. International Journal of Physical Distribution and Logistics Management, 40(8/ 9), 722. [5]. Dervitsiotis, K. (2008). Guiding human organizations to climb the spiral stages of performance improvement. Total Quality Management & Business Excellence, 19(7/8), 709-718. [6]. Domanski, B. (1994). Client-server update. Capacity Management Review, 22(7), 7. [7]. Evans, R. (1980). Basic data communications techniques. MiniMicro Systems, 13(3), 97. [8]. Friedman, D. (2008). Fifty years of Information Technology Advances. Supply House Times, 51(2), 106. [9].Hill, B. (2005). Organizational design implications. Computational & Mathematical Organization Theory, 11, 59. [10]. Hoagland, A. (2005). Magnetic disk heritage center. Business Wire (May 26), 1.
Š 2012 ACEEE DOI: 02.CSA.2012.01.4
96