Self Monitoring: A Study of Student Interactive Assessment Dean Eichorn, Science and Math Department Head, Mountain Secondary School, Langley, B. C., Canada, deichorn@sd35.bc.ca Janice Woodrow, Professor Ermerita, Department of Curriculum Studies, Faculty of Education, University of British Columbia, Vancouver, B. C., Canada Janice.Woodrow@ubc.ca
Abstract This paper describes the results of a study which examined the impact of interactive assessment practices on science teaching and learning at the secondary level. Interactive assessment, in the context of this paper, consists of students taking quizzes/tests at a computer, and receiving immediate feedback and mastery information. Situated in a longitudinal (7 year), collaborative action research project, the Technology Enhanced Secondary Science Instruction (TESSI) project, the study examined the use of interactive assessment procedures and documented the impact of these procedures from both the students’ point of view and the teachers’. The study’s results indicate that the use of interactive assessment promotes student self-monitoring, goal setting, time management, responsibility and mastery learning. Teachers report that the use of interactive assessment facilitates and supports student-centered, instructional practices.
Introduction The context for this study is the Technology Enhanced Secondary Science Instruction (TESSI) project (see Woodrow, Mayer-Smith & Pedretti 1996), a multifaceted, collaborative research endeavor between teachers and educational researchers. Initiated in 1992 in two Physics classrooms, and now expanded to ten science classrooms spanning grades 9-12, and the disciplines of Physics, Biology, Chemistry, and General Science, TESSI combines the educational opportunities provided by classroom-based, commercially-available, multimedia technologies with student-centered teaching strategies. Student learning is variably paced, individualized, collaborative, and mastery-based. TESSI’s longitudinal design has provided a platform for the critical study of the consistent use of technology by both students and teachers. This paper documents the use of interactive assessment in the TESSI classrooms. Within the context of this paper, interactive assessment consists of students taking quizzes/tests at a computer, and receiving immediate feedback and mastery information. The underlying research questions are: What happens when interactive assessment is used in a technology enhanced science classroom? What impact is there on student learning and teacher instruction? In their essay, “Does Technology Work in Schools?”, Baker, Herman and Gearhart (1996) state that “Technology use must be grounded firmly in curriculum goals, incorporated in sound instructional process, and deeply integrated with subject-matter content” if changes in teaching and learning are to be achieved. Interactive assessment effectively addresses each of these three domains. In addition, interactive assessment promises to reduce the teachers’ workload thereby freeing time for more individualized instructional practices and enable a level of tracking and monitoring individual student progress virtually impossible using traditional techniques. In the TESSI project, the focus of interactive assessment is on the use of objectives-based assessment to monitor student understanding and progress (what they know and can do), challenge and enhance student learning by providing self-regulatory and self-monitoring opportunities, and guide teacher planning and analysis of learning/instructional effectiveness. The goal is to make assessment an integral part of the instruction process, thereby enriching both teaching and learning.
Interactive Assessment in TESSI Classrooms
Page 193
TESSI uses the testing program, LXR•TEST™ for its assessment procedures. This software can create interactive tests (with or without feedback) as well as standard, paper-based tests based upon user selected key-words or objectives from stored item banks, and provide feedback on mastery to both teacher and student. Each TESSI unit includes several interactive quizzes which students complete when they judge that they have mastered the requisite material. Some TESSI teachers provide both practice and scored quizzes for their students, while others provide the option of doing some corrective work designed to help with learning the specific concepts that were not mastered at the time of the first quiz, and an opportunity, upon completion of this corrective work, of taking another quiz. All interactive quizzes are configured to provide immediate scoring and item feedback. While some students are at the computers taking quizzes, the other students remain engaged in other classroom activities. Students are also given the opportunity of completing their interactive quizzes outside of classroom time. The interactive features of LXR•TEST™ enable the setting of questions based upon QuickTime® movies of simulations, animations, or video clips, as well as digitized images of equipment and examples encountered in the learning activities. This capability of LXR•TEST™ promotes the visualization of concepts and enables the assessment of higher order objectives than is generally possible with objective tests. The end-of-unit tests are prepared and scored using LXR•TEST™ but the students write these tests traditionally—i.e. using pencil and paper—in part, to prepare them to write the Provincial exam at the end of grade 12. To facilitate the marking of the unit tests, responses to the objective questions are entered on optical mark reader (OMR) bubble sheets and scanned into the computer. Marks for free-format questions are entered into the computer manually by the teacher. As with the interactive quizzes, the LXR•Test™ Scoring Edition is used to generate test statistics and update test item statistics. Mastery reports are given to students after each quiz in order for students to examine how well they did on each objective. These mastery reports provide both the teacher and the student with information about what has been learned and help the students to study effectively for the summative, unit tests. LXR•TEST™ can generate a variety of mastery reports based on the objectives which have been used to organized the item bank or question file. For this process to generate useful information, (and to enable tests to be created quickly) the item banks must be carefully constructed to coincide with curricular objectives. TESSI teachers use item banks being collaboratively developed by the Assessment Resource Consortium (ARC). ARC is a consortium of teachers from across the Province who are developing, testing and categorizing test items geared to the Provincial learning outcomes. Each item in the ARC bank is stored via an assigned objective and a set of seven “key words”: Topic, Sub Topic 1, etc. The key words provide a rapid method of selecting subsets of questions from the item bank. These item banks are upgraded and published yearly in CD-ROM format.
Method and Data Sources A qualitative case study methodology (Yin 1994; Stake 1994; Merriam 1988) was used to examine the implementation of interactive assessment within the context of the TESSI classrooms. Techniques borrowed from the ethnographic tradition (Hammersly & Atkinson 1990) were used to document the multiple perspectives and views about interactive assessment of the TESSI teachers and students spanning a period of three years. Two major sources of data informed this study: journal entries by the ten TESSI teachers and audiotaped interviews with TESSI students during 1995-96. Supplementary data sources include informal, teacher-administered student questionnaires, focused interviews with the TESSI teachers, video tapes of meetings between TESSI teachers and the project director, field notes compiled during researcher visits to TESSI classrooms, and results of required, grade 12 Provincial exams. The study’s findings are based on the analysis of these multiple data sources.
Findings Student-centered, variable paced programs became manageable in this project because the technology takes over much of the traditional assessment role of teachers allowing them to interact in new and more effective ways with their students.” (Feedback,
Page 194
TESSI Teacher, 1998) “I couldn’t do what I do without interactive assessment!” (Journal entry, TESSI teacher, March, 1998) The above quotes, by two of the founding TESSI teachers, are persuasive statements about the value of interactive assessment. All teachers in the TESSI project regard interactive assessment as an indispensable element of their instructional practice—one which has generated many significant outcomes. Some of these outcomes are described below. Quotes from teachers and students are included to provide context, support and emphasis.
a) Self-Monitoring It is well established that young people are very anxious to achieve, generally, and, more importantly, to achieve stated goals. Yet, many young people, including those of high school age, do not see the relationship between concept attainment (i.e. understanding) and summative achievement (test marks). Students too often take a pragmatic approach to school and engage in what Fenstemacher (1994) refers to as “studenting,”—participating in the actions of completing assignments and writing tests with little regard for learning. Students’ essential focus, and often, primary goal, is just completing the job by leaping all the hurdles. In this study, it was found that most TESSI students came to regard assessment as a means of monitoring and enhancing understanding rather than as an “obstacle to be overcome.” For example, two students commented: “It gives you a greater view of how tests are like. This also can test your ability/knowledge/understanding after you learn the topic. If you seem to have wrong answers on that certain type of questions, you would know you need to go over notes/seek help.” (TESSI Chemistry 11 student, 1997) “It definitely makes the test easier if you do the practice quiz on the computer before you write the test. You know what’s on the test as well as your weak points immediately. It gives me a chance to prepare for a tests from a different point of view. It also shows me how I made my mistakes and what I can do to correct them.” (TESSI Physics 12 student) Most TESSI teachers create electronic files for students to enter their own assignment and quiz marks. This process contributes to student self-monitoring: “We keep all our assignments in a file folder and then we take them up to the computer and we put them in under the name of the assignment. It tallies up your marks so you know where you are in the class. It’s hard work but I like to know what I’m getting.” (TESSI Science 10 student, 1995)
b) Changing Student Behaviors TESSI students showed evidence of conceptual change in how they viewed their role and that of their teacher in the science classroom. They regarded themselves as directing and monitoring their own learning, and saw their teacher as a “helper” in that process. While this conceptual change is partly attributable to the nature of the TESSI instructional model and the technology enhanced learning activities in which students engage, it is also clearly related to the use of interactive assessment. For example, a teacher who was using the TESSI model for the first time this year commented that “There is not as much focus on marks—they have switched to a focus on learning. ... They are more concerned with meeting the learning outcomes (largely as a result of the mastery reports).” (Journal Entry, TESSI teacher, 1998) Since the students cannot all take quizzes at the same time with just 8-10 computers available, they are given a measure of choice as to when they complete the quizzes. The students view this opportunity to determine for themselves when they were ready to take a quiz as an important feature of their class.
Page 195
“I like [interactive quizzes] better `cause you just go and do the quiz whenever your want and if you’re not ready one day to do it, `cause you didn’t study enough, then you can wait till the next day.” (TESSI Science 10 student, 1995 ) Students have also demonstrated an increased level of responsibility and effective time management by writing quizzes on their own time. “Some students write quizzes outside of class time (which is no burden since they are on the computers and they can only write once). This helps [them] to devote more class time to learning.” (Journal Entry, TESSI Teacher, 1997) The teachers who provide practice tests for their students have found that students often work on these practice tests collaboratively. Some even begin to design their own tests. For example, two teachers write: “two or more students often work together on the practice tests which often puts an edge of competition into the learning and also makes it a co-operative learning situation.” (Feedback, TESSI Teacher, 1998) “sometimes the practice test can be 2 or 3 times as long as the real test, so the kids can actually “make up” their own different practice tests from within that like a test bank.” (Feedback, TESSI Teacher, 1998) c) Interactive Assessment as a Learning Opportunity Many students fear exam situations; they see tests as mysterious and unknown situations over which they have little or no control. Tests for these students are viewed as “end points” to the educational process, not as tools for learning. Interactive assessment, however, can become a learning opportunity for students and is one means of reversing the ingrained student perception of tests. The ability to mark questions and access solutions as the quiz is in progress was valued as a learning opportunity by nearly all TESSI students. For example, student feedback on a questionnaire querying TESSI students’ opinions about interactive assessment included the comments: “The computer marks it for you and it’s a better way of learning because when you take the quiz if you get a question wrong, it will explain it to you right there what you did wrong.” (TESSI Physics 11 student, 1996) “I like how you can mark the question right afterwards because then you learn while you are doing the test, instead of when you get it back and you have forgotten all about what you have done.” (TESSI Physics 11 student, 1996)
d) Mastery-based Learning Mastery-based learning in TESSI is managed and promoted through the teacher’s ability to diagnose specific learning problems, provide additional learning materials, and retest individual students with a minimal expenditure of time. For example, one teacher commented: “The mastery reports provide both teacher and students with detailed statistics as to strengths and weaknesses of the individual student (so that they may organize tutorials and activities with me) and of the class as a whole (weaknesses in classroom understanding means re-teach the concept).” (Interview, TESSI Teacher, March 1998) Feedback from the interactive quizzes includes information on mastery by course objectives. Teachers make this mastery information available to the students who soon learned to use it to direct their learning strategies. “I found that they love the LXR mastery reports because it gave them something positive to focus on (i.e. even though test score wasn’t high, they knew 3 of the 5 objectives cold)” (Journal entry, TESSI Teacher, 1997) “The mastery reports are extremely useful. When I go to study for a final exam or unit test I simply have to focus on those areas that I did not master.” (TESSI Physics 12 student, 1997)
Page 196
e) Changing Teacher Practices The TESSI teachers have found that the integration of student interactive assessment has impacted their classroom practice in many positive ways. Being able to use technology to produce frequent, formative assessment opportunities for their students is an efficient and reliable process that permits them to focus on other organizational and management aspects of instruction. Managing a learner-centered, technology enhanced classroom places enormous pressures on teacher time. Interactive assessment was viewed as a means of providing more “quality teaching time”: “... the interactive tests helps to free up teacher time on useless administrative tasks that the computer can easily accomplish such as marking, arranging a time for the quiz, checking the solutions, organizing a requiz - all of which can be taken care of in a one shot deal.” (Journal Entry, TESSI Teacher, 1998) “This type of testing has freed me up from time consuming photocopying, although the time spent with students going over quizzes has increased. Bottom line is contact time with students is increased.” (Journal Entry, TESSI Teacher, 1998) The TESSI teachers have also increased the frequency with which they assess their students and their assessment has become more focused on specific curricular objectives. Formative assessment, in particular, has increased partly because it has become feasible to do so. “[Interactive assessment] is probably the best aspect of TESSI for me. I'm able to quickly generate quizzes and tests that are meaningful and within curricular specifications. Students can get instant feedback and corrective solutions to their answers and I can expose them to a far greater variety of questions because I don't have the same paper concerns as with hard copies.” (Feedback, TESSI Teacher, 1998) “Using the technology also makes offering requizzes or practice tests possible and not cumbersome. Students really like having the opportunity for a requiz if they fall short of their goal.” (Journal entry, TESSI Teacher, 1997)
f) Increased Opportunities for Student Success The changes in teaching practice that interactive assessment has facilitated has provided many opportunities for increased student success. All teachers want to help their students succeed. TESSI teachers have been able to realize this goal. For example, two teachers write: “I find that the students are more prepared [after interactive assessment] for any tutoring... They are much more focused and have particularly thoughtful and relevant questions.” (Feedback, TESSI Teacher, 1998) “my good students use the interactive tests to check their understanding as they sequentially progress through the study guides and they complete the corrective assignments and try the retests when they are not successful the first time.” (Feedback, TESSI Teacher, 1998) These increased opportunities for success are not lost on the students. “You're freer to work at your pace [on individual activities]... if you take a test and do really bad on it there's a lot of extra things, work you can get to fix that. There's help if you need it.... [in] a lot of other classes you just work up to your test and then you have your test and that's it.” (TESSI Physics 11 student, 1996) The increased enrollment in, and successful completion of, the senior science courses in TESSI is a testament to the effect of these increased learning opportunities. After his first year implementing TESSI, one teacher wrote: “... last year’s [Physics] 12 class was very small (16 guys) and my over-all class average was slightly below the Provincial [average]. ... This year the class size has doubled (more kids who, normally, would never take Physics) ... I was quite worried about [the results on the Provincial Exam] since 40% of my class were C students in Ph 11 and very
Page 197
weak in math. They came out of the test feeling pretty good (some even said it was easy). The overall average was the same as the Provincial average. Furthermore, the top students did extremely well (better than last year). I had a 97%, a 96%, three with 92%, an 87%, and an 84%. Last year (with the “cream of the crop kids”) I had a 98% and an 89% (no other A’s recorded). (Journal entry, TESSI Teacher, 1998) Another wrote: “I feel that the marks in my classes have gone up by up to 10% across the board, just because of the greater opportunity I have offered to my students when it comes to exam preparation.” (Journal entry, TESSI Teacher, 1997)
g) Classroom Management The use of student interactive assessment has a major impact upon classroom management. Some of these impacts are positive. For example: “Marks are quickly imported (within 10 minutes of a quiz) to a Marks program. This insures that students have a quick idea as to their current grade and frees up my time.” (Feedback, TESSI Teacher, 1997) Some, however, require creative solutions: “it is hard to monitor what students say outside of room regarding a quiz.” (Feedback, TESSI Teacher, 1997) “Since everyone is not writing at the same time, the noise in the room can get a little high for a test situation.” (Journal entry, TESSI Teacher, 1997) “With very large classes, 28 students, trying to fit everybody in one class is a problem, especially if quizzes take more than 20 minutes. Students not taking the quiz didn't use the non-quiz time as effectively as I'd hoped. Also the class layout isn't the best for this. Cheating is a whole other potential problem!” (Journal entry, TESSI Teacher, 1998) In the school in which two TESSI classrooms are operating, students are given the privilege of using a computer in either classroom for their quizzes. Very few students abuse this privilege. Another innovative classroom management strategy is for the teacher to access the classroom fileserver from home to create quizzes or analyze results. This strategy is not without it problems: “Just downloaded a file containing student passwords from school to create an interactive test, and guess what!!? Another hard drive crash. So here I am, sick at home with my computers saying “Just try to get me working!!!” (Journal entry, TESSI Teacher, 1998)
Conclusion Teachers from the ten classrooms currently participating in the TESSI project viewed the implementation of interactive assessment as an essential and enabling element of their student-centered, variable paced, instructional practices. Students viewed interactive testing procedures as a means for assessing their own learning, frequently commenting on the usefulness of the formative feedback they received. This study supports the notion that changes in student assessment should accompany, and be congruent with, changes in educational practice, and demonstrates how this congruency can be achieved. Furthermore it shows that technology based interactive assessment can promote student goal setting, self-monitoring, time management skills, student responsibility and mastery learning—skills that will aid our graduates in being successful life-long-learners. “Interactive assessment not only allows for more understanding but also increases student confidence and provides more opportunities for remediation.” (Feedback, TESSI Teacher, 1998)
Page 198
References Baker, E. L., Herman, J. L., & Gearhart, M., (1996). Does technology work in schools? Why evaluation cannot tell the full story. In Fisher, C., Dwyer, D. C. & Yocam, K., (Ed.s) Education & Technology: Reflections on Computing in Classrooms (pp 185-202). San Francisco, CA: Apple Press, Jossey-Bass. Fenstermacher, G. D. (1994). Studenting: Promoting an institutional focus on improving college teaching. Paper presented at the annual general meeting of the American Educational Research Association, New Orleans, LA. Hammersly, M., & Atkinson, P., (1990). Ethnography Principles in Practice. London: Routledge Merriam, S. B. (1988). Case Study Research in Education: A Qualitative Approach. San Francisco, CA: Jossey Bass. Stake, R. (1994). Case studies. In N. K. Denzen & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 236-247). Thousand Oaks, CA: Sage. Woodrow, J. E. J., Mayer-Smith, J. A., & Pedretti, E., G. (1996). The impact of technology enhanced science instruction on pedagogical beliefs and practices. Journal of science education and technology, 5(3), 241-252. Yin, R. K. (1994). Case study research: Design and methods (2nd Edition). Thousand Oaks, CA: Sage.
Acknowledgments The authors would like acknowledge the support of the many agencies and firms that have provided financial and in-kind support to TESSI including the Vancouver Foundation, the Chawker’s Foundation, the Langely, Richmond, Abbotsford, Kelowna and Powell River School Districts, the University of British Columbia, Prentice Hall, Canada, Apple Computers, Canada, Logic eXtension Resources, Knowledge Revolution, Merlan Scientific, Pasco Scientific, Texas Instruments, Center for Image Processing in Education, the B. C. Ministry of Education, and the TeleLearning Research Network.
Page 199