Beyond Penguins Evaluation Report 2009-2010 by Digital Libraries @ Ohio State University

Evaluation of Beyond Penguins and Polar Bears: Integrating Literacy and IPY in the K-5 Classroom Annual Report 2009-2010

Miami University 408 McGuffey Hall Oxford, OH 45056

Phone: 513-529-1686 Fax: 513-529-2110 Website: http://ohioeval.muohio.edu

Please cite as follows: Woodruff, S. B., Morio, K. L., & Li, Y. (2010). Evaluation of beyond penguins and polar bears: Integrating literacy and IPY in the K-5 classroom. Annual report 2009-2010. Oxford, OH: Miami University, Ohioâ&#x20AC;&#x2122;s Evaluation & Assessment Center for Mathematics and Science Education. Distributed by Ohioâ&#x20AC;&#x2122;s Evaluation & Assessment Center for Mathematics and Science Education Sarah B. Woodruff, Director 408 McGuffey Hall Miami University Oxford, Ohio 45056

Evaluation of Beyond Penguins and Polar Bears

Evaluation of Beyond Penguins and Polar Bears: Integrating Literacy and IPY in the K-5 Classroom

Ohioâ&#x20AC;&#x2122;s Evaluation & Assessment Center for Mathematics and Science Education

Miami University Oxford, Ohio

Sarah B. Woodruff Yue Li Kristen Morio

Principal Investigator Senior Researcher and Statistician Research Associate

Evaluation of Beyond Penguins and Polar Bears

iii

Table of Contents Table of Contents ................................................................................................................. iv List of Tables .......................................................................................................................vii List of Figures ..................................................................................................................... viii Introduction .......................................................................................................................... 1 Project Description ............................................................................................................ 1 Evaluation............................................................................................................................. 2 Participants ....................................................................................................................... 2 Instruments ...................................................................................................................... 2 BPPB Teacher Questionnaire .......................................................................................... 3 Reliability .................................................................................................................. 3 BPPB Student Questionnaire ........................................................................................... 4 BPPB Student Questionnaire Cognitive Laboratory........................................................ 4 Reliability .................................................................................................................. 6 OCEPT â&#x20AC;&#x201C; Teacher Observation Protocol (OTOP) ............................................................... 7 Inside the Classroom Teacher Interview Protocol ............................................................. 7 Data Collection .................................................................................................................. 7 Data Analysis .................................................................................................................... 8 Findings................................................................................................................................ 9 BPPB Teacher Questionnaire, Pre (Fall 2009) and Post (Spring 2010) .................................... 9 Teacher Demographics .................................................................................................. 9 Teaching and Learning................................................................................................... 9 Integration of BPPB Materials ....................................................................................... 12 Evidence of Impact on Students ................................................................................... 13 BPPB Student Questionnaire, Pre (Fall 2009) and Post (Spring 2010) .................................. 14 Student Demographics ................................................................................................. 14 Science Views ............................................................................................................. 16 Webmetrics Analysis ........................................................................................................ 17 Unique Visitors ............................................................................................................ 17 Traffic Sources ............................................................................................................ 18 Search Keywords ......................................................................................................... 18 Content Usage ............................................................................................................ 20

Evaluation of Beyond Penguins and Polar Bears

Unique Visits to the Beyond Penguins and Polar Bears Blog ............................................ 21 Use of Digital Stories ................................................................................................... 22 Observations ................................................................................................................... 23 BPPB Field-Test Teacher Classroom Observation Scores ................................................. 23 BPPB Field-Test Teacher Classroom Observation Narratives ............................................ 25 Content Implementation .......................................................................................... 25 Literature and Science ............................................................................................. 25 Teaching Practices ................................................................................................... 25 The Beyond Penguins and Polar Bears (BPPB) Website............................................... 26 Interviews....................................................................................................................... 26 BPPB Classroom Observation Follow-up Interviews ........................................................ 26 Document Review............................................................................................................ 27 BPPB Field-Test Teacher Field Journals ......................................................................... 27 Content Knowledge.................................................................................................. 27 Literature and Science ............................................................................................. 28 Student Impact ....................................................................................................... 28 Teaching Practices ................................................................................................... 28 Beyond Penguins and Polar Bears (BBPB) Website Usage ........................................... 29 Dissemination.................................................................................................................. 30 Presentations at Local and National Conferences and Meetings ....................................... 30 Distributed Promotional Materials at Conferences and Meetings ...................................... 30 National Science Teachers Association Conference Presentation Evaluations .................... 30 National Science Teachers Association Web Seminar Evaluations .................................... 31 National Science Digital Library Brown Bag Series: Web Seminar Evaluations ................... 32 Continuing Evaluation Activities ............................................................................................ 33 Summary and Recommendations.......................................................................................... 34 References.......................................................................................................................... 36 Appendices ......................................................................................................................... 37 Appendix A. Teacher Pre-Questionnaire ................................................................................ 38 Appendix B. Teacher Post-Questionnaire ............................................................................... 42 Appendix C. Student Questionnaire ....................................................................................... 47 Appendix D. OCEPT-Teacher Observation Protocol (O-TOP).................................................... 49 Appendix E. Inside the Classroom: Teacher Interview Protocol ............................................... 51 Appendix F. BPPB Evaluation Matrix ...................................................................................... 53 Appendix G. NSTA Web Seminar Evaluation: Arctic and Antarctic Birds .................................... 54

Evaluation of Beyond Penguins and Polar Bears

Appendix H. NSTA Web Seminar Evaluation: Timely Teachings: Seasons and the Cycles of Night and Day.............................................................................................................................. 60 Appendix I. NSDL Web Seminar Evaluation: Brown Bag Series ................................................ 66

Evaluation of Beyond Penguins and Polar Bears

List of Tables Table 1. Subscale Reliability for the BPPB Teacher Pre-Questionnaire, Fall 2009 ......................... 4 Table 2. Subscale Reliability for the BPPB Teacher Post-Questionnaire, Spring 2010 ................... 4 Table 3. Reliability for the BPPB Student Questionnaire, Pre and Post, 2009-2010 ...................... 7 Table 4. Respondent Years of Teaching Experience, Teacher Pre-Questionnaire ......................... 9 Table 5. Respondent Highest Degree Earned, Teacher Pre-Questionnaire .................................. 9 Table 6. Pre-Post Comparisons for Respondents’ Instructional Activities, Teacher Questionnaire...................................................................................................................... 10 Table 7. Pre-Post Comparisons for Students’ Learning Activities, Teacher Questionnaire ........... 11 Table 8. Respondents’ Integration of BPPB Materials, Teacher Post-Questionnaire.................... 13 Table 9. Respondents’ Evidence of Impact on Students, Teacher Post-Questionnaire................ 14 Table 10. Grade 3 Respondent Student Gender and Ethnic Background, Student PreQuestionnaire...................................................................................................................... 15 Table 11. Grade 4 Respondent Student Gender and Ethnic Background, Student PreQuestionnaire...................................................................................................................... 15 Table 12. Grade 5 Respondent Student Gender and Ethnic Background, Student PreQuestionnaire...................................................................................................................... 15 Table 13. Respondent Student Age, Student Pre-Questionnaire............................................... 16 Table 14. Pre-Post Comparisons for Student Respondents’ Science Views, Student Questionnaire...................................................................................................................... 16 Table 15. Field-Test Teacher Subsample for Classroom Observations, 2009-2010..................... 23 Table 16. Observed Field-Test Teacher OTOP Scores ............................................................. 24 Table 17. Interviewed Field-Test Teacher Classroom Environment Description ......................... 26 Table 18. BPPB Website Issues Explored by Field-Test Teachers, 2009-2010 ........................... 29 Table 19. NSTA Conference Presentation Evaluation Results, March 2009 ................................ 31

Evaluation of Beyond Penguins and Polar Bears

vii

List of Figures Figure 1. Unique visitors to the BPPB website, March 2009 – March 2010. ............................... 17 Figure 2. Traffic sources for the BPPB website, March 2009 – March 2010. .............................. 18 Figure 3. Frequency of visits resulting from website name keyword searches, March 2009 – March 2010.............................................................................................................. 19 Figure 4. Frequency of visits resulting from teaching and learning keyword searches, March 2009 – March 2010.................................................................................................... 20 Figure 5. Top content pages viewed on the BPPB website, March 2009 – March 2010. ............. 21 Figure 6. Unique visitors to the BPPB blog, March 2009 – March 2010. .................................... 22 Figure 7. Views of digital stories, March 2009 – March 2010. .................................................. 23

Evaluation of Beyond Penguins and Polar Bears

viii

Introduction Ohio’s Evaluation & Assessment Center for Mathematics and Science Education (E & A Center) is the external evaluator for Beyond Penguins and Polar Bears (BPPB), a project funded by the National Science Foundation (NSF). Dr. Sarah B. Woodruff, Miami University, is the Principal Investigator (PI) for the evaluation and Kristen Morio, Research Associate, is the Project Director. Yue Li is the Senior Researcher and Statistician for the evaluation. This report is divided into four sections. Section one provides background information about the BPPB project and goals. Section two provides information and findings from Year 3 evaluation and project activities. Section three provides future evaluation plans and ongoing evaluation activities. Section four summarizes the evaluation for Year 3 and provides summative recommendations.

Project Description Beyond Penguins and Polar Bears (BPPB) is a collaboration between the Ohio State University (OSU) College of Education and Human Ecology, the Ohio Resource Center for Mathematics, Science, and Reading (ORC), the Byrd Polar Research Center, the Center of Science and Industry (COSI) Columbus, and the National Science Digital Library (NSDL). The project focuses on integrating science and literacy to educate Grades K-5 teachers on polar concepts using an electronic magazine interface. The project has four goals: •

Goal 1: Providing context to online resources by creating, identifying, selecting, and adapting quality learning resources from the National Science Digital Library, the Ohio Resource Center, other IPY-funded projects, and additional high-quality content providers.

•

Goal 2: Modifying and building communication, production, and cyberinfrastructure tools to amplify resource discovery and access to resources, increase the ease of reuse and repurposing of content, decrease production times, and increase automated dissemination of IPY materials to various audiences.

•

Goal 3: Disseminating deliverables through presentations, publications, digital libraries and push technologies.

•

Goal 4: Evaluating the impact of the project deliverables on Grades K-5 teachers and students.

Evaluation of Beyond Penguins and Polar Bears

Evaluation The overarching evaluation effort focused on assessing progress toward project goals and monitoring project implementation. During Year 1 of the evaluation, the E & A Center staff worked with the Project Team on the development and refinement of the evaluation matrix. During Year 2, evaluation activities included monitoring of website statistics, conducting expert review of project deliverables, analyzing online surveys to determine the effectiveness of the Beyond Penguins and Polar Bears (BPPB) website (http://beyondpenguins.nsdl.org), and piloting materials in Kindergarten and Grade 3 classrooms to gather formative impact data. Year 3 evaluation activities combined qualitative and quantitative data collection to assess further progress toward project goals. Primary foci of Year 3 activities included the collection and analysis of impact data from pre- and post-questionnaires. These data were collected from teachers recruited to field test BPPB deliverables as well as students taught by these teachers. Protocol-based classroom observations and interviews of field-test teachers were collected and analyzed. Webinar and conference evaluation data and webmetric reports were collected and analyzed as a continuation of Year 2 evaluation activities.

Participants The BPPB project focused on integrating science and literacy to educate Grades K-5 teachers on polar concepts using an electronic magazine interface. Therefore, teachers of Grades K-5 were recruited from Columbus City Schools (Columbus, OH) and from Charlotte Mecklenberg School District (Charlotte, NC) to field test the BPPB website. Targeted recruitment emails were sent from the Charlotte Mecklenberg Elementary Science Specialist to teachers in the Charlotte Mecklenberg School District. Recruitment information also was posted on the Ohio Department of Education (ODE) Tools for Teachers e-newsletter as well as the Teaching and Learning Collaborative: Science Works newsletter. Teachers also were recruited at science teaching conferences. Teachers recruited for field testing attended a web seminar orientation in September 2009 in which they were introduced to the BPPB website and informed of expectations as project participants. After touring the BPPB website, teachers were asked to choose a topic that would fit into their Fall 2009 curriculum. Using resources from the BPPB website, field-test teachers implemented a unit in their classrooms. Teachers were given a list of required elements to use in the field-test procedure, including: science and literacy content knowledge articles, science and literacy lessons and activities, feature story, and virtual bookshelf. Starting with the planning process, teachers were asked to keep a log tracking BPPB website use along with implementation thoughts and procedures. Selected field-test teachers were observed and interviewed to collect further data on implementation of resources. Pre- and post-data with regard to impact of the project on teachers and students were collected via questionnaires.

Instruments Evaluation data were collected in Year 3 from four sources: pre- and post-questionnaires administered to teachers and students, classroom observations, teacher interviews, and teacher field journals.

Evaluation of Beyond Penguins and Polar Bears

The BPPB Teacher Questionnaire and the BPPB Student Questionnaire were administered before and after website resource implementation to collect impact data for project evaluation. Qualitative data were collected through classroom observations, teacher interviews, and field journals. Classroom observations were performed using the OCEPT – Teacher Observation Protocol (O-TOP), (Wainright, Morrell, Flick, & Schepige, 2004). The follow-up interview protocol was developed for this project based upon the Inside the Classroom: Teacher Interview Protocol (Horizon Research Inc., 2000).

BPPB Teacher Questionnaire The BPPB Teacher Questionnaire is an adaptation of a valid and reliable instrument used in the Ohio Middle Level Mathematics and Science Education Bridging Study (Kahle & Rogg, 1996). Two of the five original subscales were used for the BPPB project, “How I Teach” and “What My Students Do.” Items were added to both subscales to reflect the project focus on literacy in the science classroom. Seven items were added to the “How I Teach” subscale and four items were added to the “What My Students Do” subscale. These additional items were developed by project staff. The E & A Center Evaluation Team modified the items and formatted the subscale. The adapted pre-questionnaire was administered to field-test teachers in Fall 2009, and it can be found in Appendix A. The teaching and learning subscales (“How I Teach” and “What My Students Do”) each consisted of 19 items on a 5-point Likert-type scale with responses ranging from almost never (1) to very often (5). The “Knowledge of Polar Regions” subscale was deleted from both pre- and postquestionnaires in Year 3. The pre-questionnaire also collected demographic information. The post-questionnaire consisted of the subscales previously noted with two additional subscales that measured impact of project materials on teacher participants and their students. These two additional subscales were: “Integration of BPPB Materials” and “Evidence of Impact on Students.” Both of these subscales consisted of 11 items and were on a 5-point Likert-type scale with responses ranging from strongly disagree (1) to strongly agree (5). The “Evidence of Impact on Students” subscale had three parallel items to the Student Questionnaire developed by the E & A Center. These items were (a) my students like science, (b) my students believe they are good at science, and (c) my students understand most of what goes on in science. The remaining items on the “Evidence of Impact on Students” subscale were created by the project staff and modified by the E & A Center Evaluation Team. The “Integration of BPPB Materials” subscale was modeled from an instrument retrieved from the Online Evaluation Resource Library (OERL) and adapted with permission to reflect the goals of this project. This 1994-1995 Evaluation Survey was originally created for a similar National Science Foundation (NSF) project to determine the impact of an integrated program on participating teachers. The post-questionnaire was converted from paper copy into an online questionnaire using Prezza Checkbox®. It was administered to teachers in Spring 2010 and can be found in Appendix B.

Reliability Cronbach’s coefficient alpha values were calculated for the BPPB Teacher Questionnaire for preand post-questionnaire respondents separately (Tables 1 and 2). Cronbach’s coefficient alpha values were equal to or higher than .80 for all the subscales for both pre- and postquestionnaires. The instrument was found to have high internal consistency.

Evaluation of Beyond Penguins and Polar Bears

Table 1. Subscale Reliability for the BPPB Teacher Pre-Questionnaire, Fall 2009 # of Items

Subscale How I Teach a

What My Students Do

n 32 30

Cronbach’s Coefficient Alpha .86 .84

Some items were reverse coded to align the project content with participant responses prior to calculating reliability.

Table 2. Subscale Reliability for the BPPB Teacher Post-Questionnaire, Spring 2010 # of Items

Subscale How I Teach

What My Students Do

Integration of BPPB Materials

Evidence of Impact on Students

n 13 8 13 13

Cronbach’s Coefficient Alpha .85 .80 .80 .90

Some items were reverse coded to align the project content with participant responses prior to calculating reliability.

BPPB Student Questionnaire The BPPB Student Questionnaire was developed in Year 2. It was an 11-item instrument originating from a subscale of the Student Questionnaire developed for the Ohio Middle Level Mathematics and Science Education Landscape Study (Kahle & Rogg, 1996). The original instrument consisted of eight items measuring students’ views regarding science. Three additional items were added to the instrument to assess student perceptions of the importance of literacy integration in science. These items were (a) reading is important in science, (b) writing is important in science, and (c) discussion is important in science. These three new items were developed by project staff and modified by the E & A Center Evaluation Team. Rating scales for all 10 items were modified by using emoticons rather than words (i.e.,  = no,  = undecided,  = yes). The revised Student Questionnaire can be found in Appendix C.

BPPB Student Questionnaire Cognitive Laboratory During Year 2 of the project, the BPPB Student Questionnaire was piloted with Kindergarten and Grade 3 students. The instrument was found to have low reliability when used with both samples (0.27 for Kindergarten; 0.56 for Grade 3). At that time, reliability coefficients were recalculated for scenarios in which each item was removed and a number of items (i.e., Items 3, 5, and 6) were found to be problematic. While not desirable, the low reliability of the instrument is consistent with the results of other researchers who have noted persistent issues with reliability of instruments administered to young children (see Pellegrini & Bjorklund, 1998). Some researchers suggest that children are particularly vulnerable to extraneous factors, making their perceptions and performance more difficult to measure reliably. Further, while readability estimates were conducted on the instrument prior to administration, a few items may have posed language issues for Kindergarten students. The results of the instrument reliability analysis indicated a need to collect more data on the use of the instrument with young students. In October 2009, an E & A Center evaluator conducted a cognitive laboratory with two small groups of students (Kindergarten, n = 4 and Grade 3, n = 4) in order to explore instrument content and face validity issues for the BPPB Student Questionnaire.1 A cognitive lab is an 1

Prior to the cognitive lab, one item, learning science is mostly memorizing facts, was deleted from the questionnaire and wording of two other items were slightly modified.

Evaluation of Beyond Penguins and Polar Bears

interview procedure that occurs during or immediately following the administration of a questionnaire. This cognitive lab was used to determine the effects of questionnaire design and language usage on the responses of students. Using cognitive labs to aid in creating questionnaires for elementary aged students can be of particularl benefit, as younger students reading and listening skills have not yet been fully mastered. Research has shown that verbalizations can provide valuable data on how readers respond to text, providing insight into the extent to which responses were based upon a misunderstanding of what was being asked. The BPPB cognitive lab procedure employed retrospective verbal report by participants. Each item on the instrument was read to students and students were provided time to respond to each item. Following the independent completion of the entire instrument, participants were asked, in a small group, by the E & A interviewer to describe their thought processes as they responded to each question. Interview questions were informal and included, “Tell me what this says using your words;” “Are there any words in this question that you do not know?” or “What did you think about when you answered this question?” The participants were students of participating BPPB teachers and all attended an urban elementary school. Students were selected randomly from those who had returned parental consent forms and each student verbally assented to the interview. The Kindergarten student group included two White females, one White male, and one African American male. All students were 5-years-old. The Grade 3 student group was composed of two White females, one Hispanic/Latina female, and one African American male. All Grade 3 students were 8-years-old. Interview notes were taken by the interviewer and notes were transcribed. Data were analyzed in order to determine whether the questionnaire had both face and content validity when used with students in Grades K and 3 and data were used to make modifications to the instrument. The following observations were used to guide modifications to and recommendations for administration of the instrument: •

Kindergarten students stated that they did not know the meaning of several words on the instrument, including discussion, understand, and important.

•

Kindergarten students demonstrated no understanding of science as a body of knowledge and could not describe it by its activities. They described science as a type of work or a job and were unsure whether they had engaged in science either in the classroom or in other settings.

•

Kindergarten students were observed pattern marking and having difficulty following from question to question in order to mark the response they desired to mark, even when they understood what the question was asking.

•

Kindergarten students did not interpret the emoticons in the context of the questionnaire items. They interpreted the emoticons very literally. For example, the frowning face meant sad, unhappy, or mad (“like when you get in trouble”), rather than simple disagreement with the item statement.

•

Though they were exceptionally cooperative, Kindergarten students were highly distracted and frequently requested that items and instructions be repeated.

•

Grade 3 students were able to define all words on the instrument but noted that “some words would be too hard for young kids.”

Evaluation of Beyond Penguins and Polar Bears

•

Grade 3 students could describe science by its activities but did not refer to science as a body of knowledge. They noted that their teacher engaged them in many science activities in the classroom.

•

Grade 3 students were attentive and had few issues following instructions and marking their desired responses. They were not observed pattern marking, though they did make extraneous marks on the paper.

•

Grade 3 students reflected frequently on how younger students might respond to items and what difficulties they might encounter. They stated that the questionnaire was “too hard for first graders and maybe for some second graders” but thought that, with some guidance, younger students might be able to respond to Items 1, 2, and 3.

•

Grade 3 students also reflected on which items were easy to understand but difficult to respond to (e.g., Items 6 and 7), noting that they “did not know enough about some things.”

Based upon the findings of this cognitive laboratory process, the recommended modifications for the instrument were regarding its administration. The findings of the lab suggested that the instrument has face and content validity with students in Grade 3 but not with students in Kindergarten. It was observed that students in Grade 3 possessed the literacy and developmental skills to complete the questionnaire with an acceptable degree of reliability, but it was not observed that those in Kindergarten were able to do so and these issues would not be resolved even if changes were made to refine the language of problematic items. The findings of this lab were congruent with those of other researchers and suggested that young students have no conceptual understanding of the nature of science and that development of this understanding begins with the recognition of the activities of science (Lederman, 1992; Lederman, Abd-ElKhalick, Bell, & Schwartz, 2002). Grade 3 students were able to identify science by its activities but did not demonstrate an understanding of the conceptual nature of science. Since the construct of science is critical to this questionnaire, it would not be appropriate to administer it to students who have not yet grasped the construct even at rudimentary levels. For these reasons, the recommendation was made to administer the questionnaire only to students of Grades 3 and above. No other modifications were made to the instrument design or items.

Reliability Cronbach’s coefficient alpha values were calculated for the student instrument for each grade level and for all grades combined. The instrument was found to have acceptable reliability (α = .59). Reliability coefficients were recalculated for pre- and post-data for scenarios in which each item was removed (see Table 3). Removing or modifying Item 5, boys are better at science than girls, would increase the instrument reliability coefficient substantially (to α = .62) for each group and for all grade levels combined.

Evaluation of Beyond Penguins and Polar Bears

Table 3. Reliability for the BPPB Student Questionnaire, Pre and Post, 2009-2010 Cronbach's Coefficient Alpha (Pre- and Post-Data Combined) Grade 3 Grade 4 Grade 5 All Grades (n = 101) (n = 40) (n = 82) (n = 266a) Science Views

.62

.72

.55

.59

Cronbach's Alpha if Item Deleted Q1

.56

.73

.52

.56

.58

.68

.48

.55

.57

.72

.51

.56

.58

.69

.55

.56

.66

.74

.58

.62

.63

.70

.50

.58

.62

.71

.58

.59

.70

.47

.56

.59

.68

.48

.55

Q10

.58

.66

.51

.55

Q4 Q5F

Data with missing grade information also were included. b Some items were reverse coded to align the project content with participant responses prior to calculating reliability.

OCEPT – Teacher Observation Protocol (OTOP) The OCEPT – Teacher Observation Protocol (O-TOP) was comprised of 10 items that were rated on an observational category scale ranging from not observed (N/O, scored as a 0) to characterizes lesson (4). The protocol items described reform-based classroom characteristics and included a focus (e.g., metacognition, divergent thinking) and descriptions of observable teacher and student behaviors. The O-TOP instrument can be found in Appendix D.

Inside the Classroom Teacher Interview Protocol The Inside the Classroom: Teacher Interview Protocol (Horizon Research Inc., 2000) was comprised of 18 multi-part interview questions and 3 questions requesting preliminary information. The questions were categorized under 5 main headings; “Learning Goals,” “Content/Topic,” “Resources Used to Design the Lesson,” “The Teacher,” and “Context.” The protocol was adapted for this project, specifically by using only categories that applied to the project; “Learning Goals” and “The Teacher.” The final adapted protocol contained 9 multi-part questions. A copy of the adapted Inside the Classroom: Teacher Interview Protocol can be found in Appendix E.

Data Collection The BPPB Teacher Pre-Questionnaire was administered in Fall 2009 before teachers started exploring the Beyond Penguins and Polar Bears (BPPB) website. Teachers of Grades 3 through 5 administered the BPPB Student Pre-Questionnaire to their students before implementing BPPB materials in the classroom. Teachers were asked to keep field journals starting at website introduction and topic selection through implementation of materials. Classroom observations were conducted using O-TOP in classrooms selected through stratified sampling between October

Evaluation of Beyond Penguins and Polar Bears

2009 and January 2010. A purposeful subsample of teachers was interviewed following observation using a modified version of Horizon Research, Inc. Inside the Classroom: Teacher Interview Protocol. After content implementation, Student Post-Questionnaires were administered. As a final task, teachers completed the online Teacher Post-Questionnaire in Spring 2010. Website statistics (webmetrics) were collected for the BPPB project evaluation starting in March 2008. Webmetrics included in this evaluation report were collected between March 2009 and March 2010.

Data Analysis The purpose of the pre-post questionnaires was to measure the impact of the BPPB content on teachers and students. Teacher data were analyzed in terms of classroom practice change by comparing pre- and post-questionnaire mean scores using paired-samples t-tests. Frequencies for “Integration of BPPB Materials” and “Evidence of Impact on Students” from teacher post-data also were reported. Student pre- and post-data were compared using paired-samples t-tests to assess students’ attitudes toward science learning before and after studying the content using BPPB materials. A significance level of p < .05 was chosen for all analyses. Qualitative analysis software, NVivo, was used to perform theory driven content analysis on interview transcripts, observation narratives, and teacher field journals. Coding was developed to correlate with project goals and activities as defined by the BPPB Evaluation Matrix (Appendix F). O-TOP observation scores were quantified and reported to assess the teaching practices used as field-test teachers implemented BPPB deliverables in the classroom. Webmetrics were analyzed for month-to-month trends in website usage during Year 3 of the project.

Evaluation of Beyond Penguins and Polar Bears

Findings BPPB Teacher Questionnaire, Pre (Fall 2009) and Post (Spring 2010)

Teacher Demographics In Fall 2009, 38 teachers responded to the Teacher Pre-Questionnaire. However, only 13 teachers completed all field-test tasks and responded to the Teacher Post-Questionnaire. Therefore, demographics provided in Tables 4 and 5 only include information from these 13 teachers. Of 13 respondent teachers, 12 were female and one was male. Two teachers identified themselves as African American while the rest indicated White. Only one teacher had a degree in a science-related discipline. Two respondents (15%) reported having had no science professional development experience, 31% reported some and 54% much. Tables 4 and 5 display years of teacher experience and highest degree earned. The majority of teachers had more than 10 years of teaching experience and held at least a Master’s degree. Table 4. Respondent Years of Teaching Experience, Teacher Pre-Questionnaire Years 0-1 2-5 6-10 11-15 Over 15 Total

n 1 0 3 5 4 13

% 8 0 23 38 31 100

Table 5. Respondent Highest Degree Earned, Teacher Pre-Questionnaire Degree Bachelor’s Master’s Master’s + 30 credit hours or more Doctorate Total

n 3 4 6 0 13

% 23 31 46 0 100

Teaching and Learning Table 6 provides results for the pre-post comparison of teacher responses on the “How I Teach” subscale. While post-questionnaire means were higher for 12 items, only one item from this subscale showed statistically significant change in the level of frequency of using inquiry-based classroom instructional practices after the use of BPPB materials. Teachers had a significantly higher post-questionnaire mean score on the item integrate science instruction with literacy by providing opportunities for reading than the pre-questionnaire mean score. Although little

Evaluation of Beyond Penguins and Polar Bears

significant change was seen on this subscale, it should be noted that pre-questionnaire mean scores were high indicating the possibility of a ceiling effect. Table 6. Pre-Post Comparisons for Respondentsâ&#x20AC;&#x2122; Instructional Activities, Teacher Questionnaire In my classroom, I â&#x20AC;Ś arrange seating to facilitate student discussion. use open-ended questions. require that my students supply evidence to support their claims. encourage questions from my students. allow my students to work at their own pace. encourage my students to explain concepts to one another. encourage my students to consider alternative explanations. discuss the work of scientists. provide time for my students to discuss subject-specific ideas among themselves. discuss experiments from the history of science. use non-traditional/authentic assessments. give lectures. integrate science instruction with literacy by providing opportunities for reading. integrate science instruction with literacy by providing opportunities for writing.

Evaluation of Beyond Penguins and Polar Bears

Pre

4.54

0.78

0.00

1.000

Post

4.54

0.66

Pre

4.23

0.73

-0.56

.584

Post

4.38

0.77

Pre

4.23

0.93

0.35

.730

Post

4.08

1.04

Pre

4.42

0.79

-1.17

.266

Post

4.75

0.45

Pre

4.08

0.86

0.25

.808

Post

4.00

0.71

Pre

4.15

0.80

-0.22

.829

Post

4.23

0.93

Pre

3.85

0.90

-0.54

.598

Post

4.08

0.95

Pre

3.62

0.96

0.00

1.000

Post

3.62

1.04

Pre

3.69

1.03

-0.33

.746

Post

3.85

1.07

Pre

3.08

1.16

0.39

.701

Post

2.92

0.79

Pre

3.92

0.86

-0.17

.866

Post

4.00

1.00

Pre

2.85

0.69

0.89

.392

Post

2.54

1.05

Pre

3.85

0.80

-2.92

.013

Post

4.54

0.66

Pre

3.77

1.01

-1.86

.088

Post

4.38

0.65

In my classroom, I … integrate science instruction with literacy by providing opportunities for discussion. choose nonfiction, science-themed articles and books for my classroom. choose nonfiction articles and books that are not science-themed for my classroom. regularly include science texts in my reading instruction. frame science concepts using gradeappropriate real world applications.

Pre

4.00

0.82

-2.13

.055

Post

4.62

0.65

Pre

4.00

0.91

-0.89

.392

Post

4.31

0.63

Pre

3.46

1.05

-1.31

.213

Post

3.92

0.95

Pre

3.54

0.78

0.00

1.000

Post

3.54

0.66

Pre

4.00

1.08

-0.37

.721

Post

4.15

0.69

Note. Significant findings (p < .05) are indicated in bold.

Table 7 provides results for pre-post comparison of teacher responses on the “What My Students Do” subscale. Teachers indicated an increase in the frequency of students engaging in inquirybased learning activities for 13 of 19 items. However, only one item from this subscale showed statistically significant change in the level of frequency of students engaging in inquiry-based classroom learning activities after the use of BPPB materials. Teachers reported their students  write to communicate and explain scientific results significantly more often following use of BPPB materials. Table 7. Pre-Post Comparisons for Students’ Learning Activities, Teacher Questionnaire In my classroom, my students … use data to justify responses to questions. argue or debate with one another about the interpretation of data. repeat experiments to confirm results. use multiple sources of information to learn. consider alternative explanations to accepted theories. design activities to test their own ideas.

Evaluation of Beyond Penguins and Polar Bears

M 3.50 3.80

SD 0.97 0.63

t -0.76

df 9

p .468

Post

n 10 10

Pre

3.00

1.05

-1.17

.273

Post

3.50

1.08

Pre

3.17

0.94

0.25

.809

Post

3.08

0.79

Pre

4.15

0.80

-1.08

.303

Post

4.46

0.66

Pre

3.42

1.08

0.29

.777

Post

3.33

0.89

Pre

2.83

1.11

-1.47

.171

Post

3.42

0.79

Pre

In my classroom, my students … consult one another as sources for learning. talk with one another to promote learning. use educational technology in class. develop scientific literacy skills. learn about real world applications of science. take notes and listen to lectures. do worksheets. learn scientific facts by using worksheets. memorize scientific facts. use science as the basis for authentic reading. use science as the basis for authentic writing. write to communicate and explain scientific results. read science-themed, non-fiction in class. Note. Significant findings (p < .05) are indicated in bold.

Pre

3.69

1.11

-1.05

.316

Post

4.08

0.86

Pre

4.31

0.85

-0.94

.367

Post

4.62

0.65

Pre

4.08

0.86

0.00

1.000

Post

4.08

0.76

Pre

3.75

0.87

-0.76

.463

Post

4.00

0.74

Pre

3.85

0.99

-0.58

.570

Post

4.08

0.76

Pre

2.45

0.82

-0.76

.465

Post

2.73

0.90

Pre

2.77

0.60

1.67

.120

Post

2.15

0.90

Pre

2.58

0.67

0.90

.389

Post

2.33

0.89

Pre

2.80

0.79

1.63

.138

Post

2.30

0.67

Pre

3.50

1.09

-0.94

.368

Post

3.83

0.72

Pre

3.46

1.05

-1.00

.337

Post

3.77

0.60

Pre

3.45

1.04

-2.32

.042

Post

4.27

0.47

Pre

3.46

1.13

-1.81

.096

Post

4.23

0.83

Integration of BPPB Materials Table 8 displays response frequencies for the “Integration of BPPB Materials” post-questionnaire subscale. All participants agreed that as a result of the BPPB project they have incorporated the BPPB content into a specific unit of study, designed activities for their students that use the objectives of the BPPB project, increased their own science content knowledge, and become more aware of changes taking place in the polar regions. However, most teachers disagreed or remained neutral on statements indicating they have changed the ways they assess student understanding and changed the science curriculum in their classrooms.

Evaluation of Beyond Penguins and Polar Bears

Table 8. Respondents’ Integration of BPPB Materials, Teacher Post-Questionnaire

Item

As a result of the “Beyond Penguins and Polar Bears” Project, I have…

Strongly Disagree & Disagree

Undecided

Agree & Strongly Agree

incorporated the BPPB content into a specific unit of study.

100

designed activities for my students that use the objectives of the BPPB project.

100

changed the instructional strategies that I use in my classroom.

changed the ways I assess student understanding.

changed the ways I use educational technology.

changed the science curriculum in my classroom.

increased my own science content knowledge.

100

gained confidence in teaching science to my students.

learned that the Arctic and Antarctica are important geographic regions.

become more aware of changes taking place in the polar regions.

100

become an advocate for protection of the polar regions.

Note. The responses strongly disagree and disagree were combined, and strongly agree and agree were combined. Note. Responses of 60% or more are indicated in bold.

Evidence of Impact on Students Post-questionnaire frequencies for the “Evidence of Impact on Students” subscale are displayed in Table 9. The majority of teacher participants (from 77% to 100%) indicated that they agreed or strongly agreed with all items on the subscale.

Evaluation of Beyond Penguins and Polar Bears

Table 9. Respondents’ Evidence of Impact on Students, Teacher Post-Questionnaire

Item

As a result of the “Beyond Penguins and Polar Bears” Project…

Strongly Disagree & Disagree

Undecided

Agree & Strongly Agree

my students like science more.

my students believe they are good at science.

my students understand more of what goes on in science.

my students have increased their knowledge of specific science concepts through the BPPB materials.

100

my students have learned the BPPB content.

my students’ scientific literacy has increased.

my students’ achievement/performance in science has improved.

my students’ effort in science has improved.

my students’ reading skills in science have improved.

my students’ writing skills in science have improved.

my students’ discussion skills in science have improved.

Note. The responses strongly disagree and disagree were combined, and strongly agree and agree were combined. Note. Responses of 60% or more are indicated in bold.

BPPB Student Questionnaire, Pre (Fall 2009) and Post (Spring 2010)

Student Demographics Tables 10 through 13 display demographic characteristics for the BPPB student respondents. Student respondents included 59 Grade 3 students, 22 Grade 4 students, and 42 Grade 5 students. A slight majority of the students in all three grade levels were female. The majority of the students in Grades 3 and 4 identified themselves as White (61% and 55%). Grade 5 had a smaller percentage of students identifying themselves as White (41%) and a larger percentage identifying themselves and African American (28%). Most students were between 8 and 11years-old.

Evaluation of Beyond Penguins and Polar Bears

Table 10. Grade 3 Respondent Student Gender and Ethnic Background, Student PreQuestionnaire Background African American American Indian/ Alaskan Native Asian or Pacific Islander Latino(a) White Other Total

Female

Male

Total

n 7 1 1 4 16 1 30

n 4 0 1 1 20 3 29

n 11 1 2 5 36 4 59

Table 11. Grade 4 Respondent Student Gender and Ethnic Background, Student PreQuestionnaire Background African American American Indian/ Alaskan Native Asian or Pacific Islander Latino(a) White Other Total

Female

Male

Total

n 2 0 0 0 7 3 12

n 3 0 0 1 5 1 10

n 5 0 0 1 12 4 22

Table 12. Grade 5 Respondent Student Gender and Ethnic Background, Student PreQuestionnaire Background African American American Indian/ Alaskan Native Asian or Pacific Islander Latino(a) White Other Total

Female

Male

Total

n 5 1 2 1 9 3 21

n 6 0 1 1 7 3 18

n 11 1 3 2 16 6 39

Note. Some students did not respond to this item.

Evaluation of Beyond Penguins and Polar Bears

Table 13. Respondent Student Age, Student Pre-Questionnaire Age 7 8 9 10 11 Total

n 1 47 31 33 11 123

% 1 38 25 27 9 100

Science Views As shown in Table 14, student pre-post comparisons indicated that respondent students (Grades 3-5) had significantly more agreement on four items after exposure to teacher instruction, utilizing the BPPB resources. Paired t-test comparisons by grade level also were performed (data not displayed). One item, almost all people use science in their jobs, was the only item that showed significant differences between pre- and post-questionnaire means for Grades 4 and 5. In addition to this item, Grade 3 students were significantly more likely to agree with the statements I like science, I am good at science, and I understand most of what goes on in science after BPPB activities. Table 14. Pre-Post Comparisons for Student Respondentsâ&#x20AC;&#x2122; Science Views, Student Questionnaire

M 2.71 2.82 2.32 2.46

SD 0.52 0.41 0.60 0.64

t -2.08

df 89

p .041

-2.03

.045

Post

n 90 90 90 90

Pre

2.77

0.54

-0.69

.494

Post

2.80

0.48

Pre

2.54

0.64

-2.07

.042

Post

2.69

0.51

Pre

1.56

0.78

0.63

.530

Post

1.52

0.75

Pre

2.18

0.65

-4.05

< .001

Post

2.49

0.59

Pre

2.40

0.66

0.00

1.000

Post

2.40

0.61

Pre

2.67

0.56

-1.29

.200

Post

2.77

0.50

Item I like science.

Pre Post

I am good at science.

Pre

I want to learn more science.

I understand most of what goes on in science. Boys are better at science than girls. Almost all people use science in their jobs. Science is useful for solving everyday problems. Reading is important in science.

Evaluation of Beyond Penguins and Polar Bears

Pre

2.71

0.52

-0.96

.339

Post

2.77

0.45

Pre

2.75

0.51

-0.54

.593

Post

2.79

0.46

Item Writing is important in science. Discussion is important in science.

Note. Significant findings (p < .05) are indicated in bold.

Webmetrics Analysis

Unique Visitors Web statistics were collected for the Year 3 evaluation of the BPPB website starting March 1, 2009. Data available in the webmetrics reports included how visitors found the website (traffic sources), keywords used to find the site (search keywords), how long each visitor remained on the website (time on website), and how many visits to the website were from unique visitors. Figure 1 shows a significant decrease in unique visitors to the website between March 2009 and August 2009, and a general increase from August 2009 to March 2010. The decreases described by the data might be attributed to the cycle of the academic year. Fewer teachers search for resource materials during summer and winter breaks.

Figure 1. Unique visitors to the BPPB website, March 2009 â&#x20AC;&#x201C; March 2010.

Evaluation of Beyond Penguins and Polar Bears

Traffic Sources Search engines were the primary source of traffic for the BPPB website (Figure 2). Referring websites made up the second largest source of traffic followed by direct visits. Google Images websites (e.g., http://images.google.com, http://images.google.co.uk) are the primary referring website with a total of over 6,000 visits. National Science Digital Library (NSDL) webpages (http://expertvoices.nsdl.org, http://nsdl.org) also referred a large number of visitors to the BPPB website accounting for 1,669 visits. It should be noted, however, that although counted by webmetrics as referral traffic, Google Images is technically a search engine and NSDL is BPPB’s host website. The number of visits from the next five referring websites were significantly fewer, accounting for between 200-250 visits each. These referring sites were: http://www.internet4classrooms.com, http://bprc.osu.edu, http://www3.telus.net, http://sldirectory.com, and http://globalwarmingkids.net.

Figure 2. Traffic sources for the BPPB website, March 2009 – March 2010.

Search Keywords Webmetrics also reported top search engine keywords used by visitors to find the Beyond Penguins and Polar Bears website. Each month the number of visits for each keyword search are counted and the top 15 keywords are listed. Variations of the website name (e.g., “penguins and polar bears,” “beyond penguins”) made up the majority of keywords used between March 2009 and March 2010. Figure 3 shows the trend in visits resulting from the website name keyword search during this time period. There was a significant increase in visits generated from website name keyword searches during the winter months (October 2009 to December 2009) when

Evaluation of Beyond Penguins and Polar Bears

teaching polar concepts could be a popular topic. Another explanation for this increase can be found in dissemination efforts (see Dissemination). BPPB Project Team members disseminated project information at three large events in November 2009. To assess resource discovery among the target audience, teaching and learning terms, including “teaching,” “lessons,” “literature/literacy,” “activities,” “classroom,” and “misconceptions” or a specific grade level also were counted. The number of keyword searches using these terms increased substantially in September 2009 and remained at high levels through March 2010 (Figure 4). Other common keywords included “polar or arctic plants,” variations of “polar arts and crafts,” and “rocks and minerals.”

Figure 3. Frequency of visits resulting from website name keyword searches, March 2009 – March 2010.

Evaluation of Beyond Penguins and Polar Bears

Figure 4. Frequency of visits resulting from teaching and learning keyword searches, March 2009 â&#x20AC;&#x201C; March 2010.

Content Usage The number of visits to content pages on the BPPB website were tracked by webmetrics for Year 3 from March 2009 to March 2010. The top two content pages viewed were tundra content knowledge and rocks and minerals lessons (Figure 5).

Evaluation of Beyond Penguins and Polar Bears

Figure 5. Top content pages viewed on the BPPB website, March 2009 â&#x20AC;&#x201C; March 2010.

Unique Visits to the Beyond Penguins and Polar Bears Blog The BPPB first blog (http://expertvoices.nsdl.org/polar) entry was dated October 11, 2007. Unique visitors to the blog between March 2009 and March 2010 are displayed in Figure 6. The monthly pattern of unique visitors to the BPPB blog is similar to the pattern of unique visitors to the BPPB website (Figure 1). There was a significant decrease from March 2009 to July 2009 followed by a steady increase up to January 2010. A significant drop in visits occurred again in February 2010 but was followed by a rebound in March 2010.

Evaluation of Beyond Penguins and Polar Bears

Figure 6. Unique visitors to the BPPB blog, March 2009 – March 2010.

Use of Digital Stories As a part of the BPPB blog, two videos called digital stories were created (“Braving the Wilderness: Developing a Polar Sense of Place” and “Antarctic Rocks: Telling Polar Stories”). The number of views was followed through the collection of webmetrics. The videos were available through both YouTube and TeacherTube.2 Figure 7 displays the trend of digital story views between March 2009 and March 2010.3 The two digital stories were viewed on TeacherTube ten times more often than on YouTube. The number of views increased from 2,475 to 4,085 on TeacherTube between March 2009 and March 2010. The number of views remained at relatively the same level on YouTube for the same time period.

YouTube and TeacherTube are websites on which videos can be shared between users. TeacherTube videos and content are geared toward educational use and are typically accessible online in schools where many websites, such as YouTube, are blocked. 3 Digital story webmetrics data were not available for February 2010.

Evaluation of Beyond Penguins and Polar Bears

Figure 7. Views of digital stories, March 2009 â&#x20AC;&#x201C; March 2010.

Observations A subsample of field-test teachers was selected through stratified sampling for classroom observations. The field-test teacher sample was distributed by grade level and then randomly selected within the grade level strata. A total of nine teachers were chosen using this method (Table 15). Arrangements were made for six observations; three selected teachers did not respond to invitations to participate in observations. Table 15. Field-Test Teacher Subsample for Classroom Observations, 2009-2010 Grades K-2 3 4 5 Total

Charlotte Mecklenberg 2 1 0 1 4

Columbus City Schools 0 2 2 1 5

Total 2 3 2 2 9

BPPB Field-Test Teacher Classroom Observation Scores A BPPB Project Team member observed teachers using OTOP between October 2009 and January 2010. During one observed lesson for each teacher, 10 reform-based classroom characteristics were scored on a scale ranging from not observed (N/O, scored as 0) to characterizes lesson scored as a (4). Table 16 displays the scores by item and by teacher.

Evaluation of Beyond Penguins and Polar Bears

Table 16. Observed Field-Test Teacher OTOP Scores

Item This lesson encouraged students to seek and value various modes of investigation or problem solving. Teacher encouraged students to be reflective about their learning. Interactions reflected collaborative working relationships and productive discourse among students and between teacher/instructor and students. Intellectual rigor, constructive criticism, and the challenging of ideas were valued. The instructional strategies and activities probed studentsâ&#x20AC;&#x2122; existing knowledge and preconceptions. The lesson promoted strongly coherent conceptual understanding in the context of clear learning goals. Students were encouraged to generate conjectures, alternative solution strategies, and ways of interpreting evidence. Appropriate connections were made between content and other curricular areas. The teacher/instructor had a solid grasp of the subject matter content and how to teach it. The teacher/instructor used various means to represent concepts. Teacher Mean a

Teacher 1 Grade 5

Teacher 2 Grade 4

Teacher 3 Grade 5

Teacher 4 Grade 2

Teacher 5 Grade 2

Teacher 6 Grade 3

Item Mean

2.83

1.83

4.00a

3.00

2.83

2.17

1.17

3.00

3.17

1.2

3.2

2.5

2.1

3.7

3.1

Mean based upon 5 available item scores.

Evaluation of Beyond Penguins and Polar Bears

BPPB Field-Test Teacher Classroom Observation Narratives Theory driven content analysis was performed on classroom observation narratives that were included as a supplement to the OTOP data. The narratives were provided by the Project Team member who observed the classrooms. NVivo was utilized by the E & A Center evaluator to analyze the qualitative data. Themes chosen to guide analyses, based upon project goals and activities, included content implementation, content knowledge, literature and science, student impact, teaching practices, and the BPPB website. Analysis of classroom observation narratives did not reveal data related to student impact.

Content Implementation Content implementation observations revealed use of topics from the BPPB website that were relevant to the classrooms of field-test teachers. Lessons observed were based upon resources from the website and covered topics such as animal habitats, animal adaptations, and biomes.

Content Knowledge Two common misconceptions were revealed during classroom observations. One teacher erroneously mentioned, “the polar regions do not have the seasons Spring and Fall.” Another teacher mixed books about Arctic and Antarctic animals without clarifying that they are from separate regions. However, another field-test teacher “reminded students that penguins were not mammals because they don’t lay eggs and that they live in Antarctica, not the Arctic.”

Literature and Science Five of the six teachers used some form of literature during the observed polar science lesson. One teacher used and created ABC books in a Kindergarten classroom. One teacher displayed multiple books from the BPPB website virtual bookshelf4 relating to the topic of the lesson, while two other teachers described the use of trade books. Kindergarten teachers read aloud to the class and/or played an electronic book.5 Grade 4 and Grade 5 students read individually or out loud as a class. In each instance of reading, materials used were either resources available from the BPPB website or books listed on the BPPB virtual bookshelf. A class of Grade 5 students discussed the type of story they were reading (fiction, non-fiction, science fiction), and a Kindergarten class was encouraged to imagine and share their own stories.

Teaching Practices For the purposes of this analysis, the topic of teaching practices represented the activities and methodologies used by field-test teachers to convey content to students. Teacher observation narratives described four teaching practices used during observed lessons. Five teachers were observed using discussion as one of the ways content was delivered to students. Two teachers modeled behaviors or activities that were then asked of the students. Observation narratives also revealed three of the six teachers used questioning techniques that promoted further investigation. Lastly student activities and outputs were collected. Five teachers included some sort of class presentation or sharing. One teacher had students collect data. Three teachers had students create graphic organizers during the observed lesson. Two teachers used hands-on activities during their observed lesson and two other teachers had students use science notebooks. Three teachers had students write during the science lesson. 4

The BPPB website Virtual Bookshelf provides a list of recommended children's books that reflect the theme of the issue and offers ideas on how to integrate them across the curriculum. 5 The BPPB website, in conjunction with http://www.contentclips.com, supplies electronic versions of feature articles called electronic books (e-books).

Evaluation of Beyond Penguins and Polar Bears

The Beyond Penguins and Polar Bears (BPPB) Website All six teachers were observed using BPPB website resources. Resources used included electronic books, graphic organizers, virtual bookshelf suggestions, and content information.

Interviews Two field-test teachers were purposefully chosen to be interviewed as a follow-up to classroom observations. The BPPB Project Team member that performed classroom observations interviewed the teachers in January 2010 using a modified version of the Horizon Research, Inc. Inside the Classroom: Teacher Interview Protocol. Interviews were transcribed and summarized for comparison.

BPPB Classroom Observation Follow-up Interviews The interviewed teachers were asked to provide some descriptive information about the classroom environments that were observed. Both teachers described their classrooms as typical of the entire grade level within their schools. Table 17 displays classroom environment descriptions. Table 17. Interviewed Field-Test Teacher Classroom Environment Description Classroom Description OTOP Observation Score Grade Students with Disabilities ESL Students Absenteeism Lesson/Activity Observed Teaching Experience

Teacher 1 1.2 5 1 5 2 students absent 2-4x/wk Food chains/biomes 20 yrs

Teacher 2 3.2 4 3 0 Not an issue Glaciers/landmasses 9 yrs

When asked to describe the unit for the observed lesson, Teacher 1 described briefly the previous lessons on food chains and biomes. She indicated the purpose of the observed lesson was to further students’ understanding of one specific biome, tundra, and to create a food chain from that biome. Teacher 2 described previous lessons on landforms in Ohio and then described the class glacier-building project and research time conducted at the school computer lab. The observed lesson was a culmination of this information using picture books, video, and graphic organizers. Teacher 1 indicated that the students enjoyed the lesson and “…linked to the booklet with the pictures” when discussing the content. The students in Teacher 2’s classroom “really understood landmasses and what that really means… and how they [animals] survive in different habitats.” She indicated that the resources and activities “…helped synthesize the information they had been given a little bit more.” Both teachers stated that they enjoyed teaching the chosen topic. The teachers were asked to describe their content knowledge, preparation, and comfort with the topic. Teacher 1 indicated that she felt “very prepared” to teach the content while Teacher 2 felt that she had to “… do a lot of research and re-read books because it was hard for me to understand at first….” Neither teacher had taught the lesson before. Teacher 2 stated that the teaching strategies used in the BPPB materials were similar to her typical teaching methodology in the alternative school environment, “being an informal school it is a perfect fit… research, hands-on, books… that’s really how we teach….” When asked about specific planning for the lesson, Teacher 1 indicated that for the chosen topic “…similar type questions in the past have

Evaluation of Beyond Penguins and Polar Bears

been on the Ohio achievement tests and so I thought that this might be a good mix to help them in case it’s on there again this year.” Teacher 2 indicated that she considered her particular group of students and “…just tried to make it basic for them enough where they could run with it even more so I just thought about their skills and interests….” Transcript analysis indicated some differences that could explain the disparity in OTOP classroom observation scores between the two teachers. Teacher 1 used the BPPB materials to guide a topic driven lesson with the goal of preparing students for items typically found on the state test. The students had little background knowledge on the topic making the information seem somewhat out of context. Evidence of this is suggested by Teacher 1’s statement, “…my students have trouble transferring information to other situations….” Teacher 2, on the other hand, used the BPPB materials in the middle of a comprehensive unit in which students were making connections between information they had learned in previous lessons and activities (Ohio landforms and glaciers) and information about places unfamiliar to them (Antarctica and the Arctic). The unit was connected with previous learning allowing students to make their own observations about topics that may be covered by state testing (habitats). Three descriptions of classroom environment also could play a role in observation differences; specifically number of students for which English is their second language (ESL), absenteeism, and teacher experience though it is unclear precisely what impact these differences may have had on teacher instructional practice and use of BPPB materials.

Document Review

BPPB Field-Test Teacher Field Journals Field-test teachers were asked to keep a field journal during BPPB website research, lesson planning, and implementation. Eleven field journals were collected and analyzed. Theory driven content analysis was performed on teacher journal entries. NVivo was utilized by the E & A Center evaluator to collect qualitative and quantitative data. Themes chosen to guide analyses, based upon project goals and activities, included content knowledge, literature and science, student impact, teaching practices, and the BPPB website.

Content Knowledge Four teachers discussed ways in which their own content knowledge was improved during the field test experience. Teachers were inspired to increase their knowledge of the polar regions using the BPPB website along with outside resources. One teacher comment revealed this inspiration saying, “I began to log on to a radio station, KNOM, in Nome, AK just to understand the amount of daylight and the weather.” Three teachers mentioned misconceptions in their journal entries. Although one teacher used the “Misconceptions” link for an issue on polar mammals, she seemed to have misunderstood and could perpetuate a misconception stating, “I also didn’t want them to think that there are only polar bears in Antarctica. This section really gave me insight on things to say and talk about with the students.” Six teachers reported on their experience fitting BPPB materials with standards or curriculum pacing guides. For instance, one teacher stated, “I looked over each of the nine units and thought about which science objectives at each grade level would be covered in each article.” Teachers also recorded notes regarding student content knowledge in terms of what their students knew, what they didn’t know, and the knowledge they attained through material implementation. Using a prior knowledge graphic organizer available from the website, one teacher recorded that her students “…did know quite a bit about what regions penguins and polar bears lived in, as well as Santa. More background knowledge about the region is needed.”

Evaluation of Beyond Penguins and Polar Bears

Literature and Science Teachers also discussed combining literature and science in their field journals. Six teachers made a total of 17 comments about teaching science and literature together. Many comments were specific to implementation. For example one teacher stated, “The children will read the novel, Mr. Popper’s Penguins, and we will use the information about researchers in the Polar Regions.” Nine comments on literature and science, however, focused on the effects of combining the disciplines in the classroom. One teacher stated, “Integrating the science unit with our literacy unit kept the focus consistent for my students and created smoother transitions.”

Student Impact Many comments recorded indicated the impact of BPPB materials on students. Eight teachers commented a total of 30 times regarding how their students reacted to the materials both in terms of engagement and academic improvement. After implementing a BPPB lesson, one teacher stated, “Their observation skills were definitely enhanced and they have referred back to this lesson as we have discussed animals in our own neighborhood.” Another teacher indicated increased engagement stating, “At the end, I asked them how they liked doing science activities and learning about science. Some of them said ‘this was science?’ They were having so much fun it didn’t seem like work to them.” Another stated, “The engagement of my students was incredible. I’ve always known that animals are very interesting for younger children to learn about and this unit made me aware of how interesting animals remain for older students as well.”

Teaching Practices For the purposes of this analysis, the topic of teaching practices represented the activities and methodologies used by field-test teachers to convey content to their students. A large category within this topic, student activities, focused on specific activities that teachers employed to engage students with the content. Six teachers commented on the practice of connecting ideas. During implementation, field-test teachers noted that students were able to make a connection to prior knowledge, between related concepts, or to real-life versions of classroom simulations. For example, one teacher discussed prior knowledge, stating: The students had prior knowledge of penguins and polar bears, but had never seen or heard of tundra. They knew very little information about Antarctica. The pictures and articles we discussed made them really feel connected to the content and feel like they were really learning something new. One teacher commented on differentiation through a BPPB resource, stating, “In our initial reading I used the upper level story. I used the other story for my lower level readers to ensure their understanding and success.” Five teachers indicated the use of discussion with their students, while three indicated the use of modeling. Nine different student activities were noted in field-test teacher journals. The use of graphic organizers was the student activity noted most frequently in field journals. Graphic organizers used included “Know, Want to know, Learned” (KWL) charts, compare/contrast charts, Venn diagrams, and other prior knowledge graphic organizers. Eight teachers commented on their use of graphic organizers. Six teachers mentioned student use of computers during BPPB resource implementation. Students used computers to access the BPPB website and resources, along with other websites to participate in activities and gather information. In fact, four teachers indicated that their students used computer labs to do research. Field journal comments from five teachers indicated the use of hands-on activities such as, “I’ve decided to use the investigations lessons

Evaluation of Beyond Penguins and Polar Bears

and have my students make the weather instruments.” Other student activities discussed included in class reading, presentations and sharing, science notebooks, and writing activities.

Beyond Penguins and Polar Bears (BBPB) Website Usage Field-test teachers made notes in their field journals regarding different issues and resources used from the BPPB website. Table 18 displays reported issue usage and the features of the issue used. Issue 10 (Polar Mammals) and Issue 1 (Sense of Place) were the top two issues used by this group of teachers. Table 18. BPPB Website Issues Explored by Field-Test Teachers, 2009-2010 Issue

Number of Teachers

Content Used Expedition to the Poles Activity; Science Content Knowledge; Literacy Content Knowledge; Science and Literacy Lessons and Activities; Feature Story

Issue 1: Sense of Place

Issue 3: Polar Patterns: Day, Night and Seasons

Issue 4: Weather and Climate

Investigations Lessons

Issue 5: Water, Ice, and Snow

Literacy Content Knowledge; Feature Story, Floating Ice Electronic Book; Science and Literacy Lessons and Activities

Issue 6: Rocks and Minerals

Information search

Issue 9: Earth’s Changing Surface

Information search

Gliding Glaciers Activity

Issue 10: Polar Mammals

Feature Story, White Wolf Electronic Book; Virtual Bookshelf; Literacy Content Knowledge; Science Content Knowledge; Science and Literacy Lessons and Activities

Issue 12: Polar Plants

Information search

Issue 13: Tundra: Life in the Polar Extremes

Issue 14: Polar Oceans

Feature Story; Virtual Bookshelf Information search

Field-test teachers also made comments regarding their own use of technology and regarding features of the BPPB website. Two teachers commented that through BPPB they attended their first webinar. Two teacher comments revealed relief that they were able to use the technologyfocused resources. One teacher noted, “I was afraid I couldn’t navigate the computer well enough, but it is user friendly. Whew!” Eight teachers total made similar positive comments about the BPPB website. Four teachers commented on using the site in the future, three commented on recommending it to colleagues, and others made specific comments about site

Evaluation of Beyond Penguins and Polar Bears

features such as, “I like the way the science and literacy content is explained for the teacher…” and “the visuals are great….”

Dissemination Project Team members disseminated project information at conferences, meetings and electronic events, including the National Science Teachers Association (NSTA) and National Science Digital Library (NSDL) collaborative web seminar series. The following is a list of dissemination activities:

Presentations at Local and National Conferences and Meetings • • •

• • • • • • • • • • •

NSTA National Conference (New Orleans) (3/18 – 3/22/09) Ecosystems: Life in the Polar Extremes Web Seminar (NSDL Brown Bag Series) (4/7/09) Arctic and Antarctic Birds Web Seminar (NSDL/NSTA Seminar Series) (4/21/09) Timely Teachings: Seasons and the Cycles of Day and Night Web Seminar (NSDL/NSTA Seminar Series) (4/21/09)  ITSCO/TechColumbus Education Innovation Mixer (Columbus, OH) (5/6/09) Monthly Tapped in Events (5/7/09; 6/4/09; 7/8/09; 8/4/09) National Education Computing Conference (NECC) (Washington, DC) (6/28 – 7/1/09) Center of Science and Industry (COSI) Teacher Resource Fair (Columbus, OH) (8/4/09) National Middle School Association Conference (Indianapolis, IN) (11/5 – 11/7/09) DR K-12 Meeting (Washington, DC) (11/8-11/10/09) National Science Digital Library Annual Meeting (Washington, DC) (11/17 – 11/19/09) Presidential Awards for Excellence in Mathematics and Science Teaching (PAEMST) (Washington, DC) (1/5/10) Ohio Educational Technology Conference (Columbus, OH) (2/3 – 2/4/10) Science Education Council of Ohio Conference (Columbus, OH) (2/23 – 2/24/10)

Distributed Promotional Materials at Conferences and Meetings • • • • • • • • • •

NSTA National Conference (New Orleans, LA) (3/18 – 3/22/09) Ohio Conference of Teachers of English Language Arts (OCTELA) (Columbus, OH) (3/27 – 3/28/09) National Education Computing Conference (NECC) (Washington, DC) (6/28 – 7/1/09) Center of Science and Industry (COSI) Teacher Resource Fair (Columbus, OH) (8/4/09) National Middle School Association Conference (Indianapolis, IN) (11/5 – 11/7/09) DR K-12 Meeting (Washington, DC) (11/8 – 11/10/09) National Science Digital Library Annual Meeting (Washington, DC) (11/17 – 11/19/09) Presidential Awards for Excellence in Mathematics and Science Teaching (PAEMST) (Washington, DC) (1/5/10) Ohio Educational Technology Conference (Columbus, OH) (2/3 – 2/4/10) Science Education Council of Ohio Conference (Columbus, OH) (2/23 – 2/24/10)

National Science Teachers Association Conference Presentation Evaluations Beyond Penguins and Polar Bears Project Team members presented at the NSTA national conference in New Orleans, LA (two presentations, 3/21 and 3/22/09). Participants at these presentations responded to a 9-item questionnaire regarding the presentation, the speaker, and the topic. This questionnaire was provided by NSTA conference organizers and administered to conference attendees after each conference session. Responses were on a 5-point Likert-type

Evaluation of Beyond Penguins and Polar Bears

scale with responses ranging from strongly disagree (1) to strongly agree (5). Evaluations for presentations by BPPB Project Team members at each regional conference are displayed in Table 19. Table 19. NSTA Conference Presentation Evaluation Results, March 2009 Mean New Orleans Session 1 (n = 5)

Mean New Orleans Session 2 (n = 4)

Overall Mean (n = 9)

I selected this session for immediate use in the classroom.

4.5

4.50

I selected this session based on the reputation of the speaker.

4.2

2.2

3.20

I selected this session to improve my personal pedagogical knowledge/skill.

5.00

I selected this session to improve my science content knowledge.

4.50

This session met my needs.

4.7

4.85

The information presented was clear and well-organized.

4.5

4.75

Safe practices were employed.

4.5

4.75

The session avoided commercial solicitation.

4.3

4.65

The session should be repeated at another NSTA conference.

5.00

4.9

4.3

4.58

Evaluation Questions

Overall Average

National Science Teachers Association Web Seminar Evaluations Jessica Fries-Gaither, a BPPB Project Team member from Ohio State University, presented two web seminars (webinars) through the NSTA website. One webinar included two co-presenters from Cornell University, Colleen McLinn (Macaulay Library of Cornell Lab of Ornithology) and Jennifer Fee (Cornell Lab of Ornithology). The NSTA invited members to complete a common evaluation form for both of the web seminars. The evaluation included four items on the structure and content of the seminar, four items rating each presenter and four open-response items. The scaled items ranged from poor (1) to excellent (5). Participants received a one-year subscription to an NSTA SciGuide for completing the evaluation. NSTA staff summarized the evaluations and provided presenters with the results. The NSTA-reported results for the two webinars (4/21/09 and 12/10/09) are described. Full summaries can be found in Appendices G and H respectively.

Arctic and Antarctic Birds: April 21, 2009 Thirty-five participants attended the Arctic and Antarctic Birds webinar presented by Jessica Fries-Gaither, Colleen McLinn, and Jennifer Fee. Thirty-one of these participants responded to a

Evaluation of Beyond Penguins and Polar Bears

survey and rated aspects of the webinar on a scale ranging from poor (1) to excellent (5). Overall, respondents rated the webinar as valuable (4.58), relevant (4.48) and interactive (4.74). The respondents agreed that the web seminar time fit their schedules (4.55). All respondents agreed they would like to see similar webinars offered in the future. The respondents found the content useful for providing resources for K-12 and pre-service teachers that model inquiry-based technology rich lessons, and for ideas on using bird topics as a teaching tool. Overall comments included an appreciation of the content and resources provided. The presenters were rated on the same scale as the webinar (poor = 1, excellent = 5) on subject knowledge, clarity, responsiveness, and pace. None of the presenters received an average score lower than 4.71. One respondent commented â&#x20AC;&#x153;this web seminar was easy for all levels of science teachers to not only understand the information but great ideas for future application of information.â&#x20AC;? One respondent recommended adding suggestions on how to modify the resources for middle and high school students. For a full summary of the Arctic and Antarctic Birds webinar evaluation report, see Appendix G.

Timely Teachings: Seasons and Cycles of Night and Day: December 10, 2009 Sixty-six participants attended the Timely Teachings: Seasons and Cycles of Night and Day webinar presented by Jessica Fries-Gaither. Fifty-three participants responded to the webinar evaluation. Overall, respondents rated this webinar as valuable (4.36), relevant (4.38) and interactive (4.51). Respondents agreed that the webinar time fit their schedules (4.26). Most respondents (98%) agreed they would like to see similar webinars offered in the future. The respondents commented that the webinar content was valuable and relevant to them, because it covered common misconceptions, clarified season and solar system issues, and provided resources for a common elementary and middle school topic. According to open-response questions, respondents appreciated resources and strategies presented. The presenter was rated on knowledge, clarity, responsiveness and pace, and did not receive an average score lower than 4.72. For a full summary of the Timely Teachings: Seasons and Cycles of Night and Day webinar evaluation report, see Appendix H.

National Science Digital Library Brown Bag Series: Web Seminar Evaluations Jessica Fries-Gaither, presented one webinar session (4/27/09) as part of the NSDL Brown Bag Webinar Series. NSDL provided a common online feedback questionnaire to evaluate the session and the presenter. The questionnaire included one item on a scale ranging from poor (1) to excellent (5) and four open-response items. The evaluation was summarized by NSDL staff and provided to the presenter. A full version of the NSDL evaluation summary is included in Appendix I.

Life in the Polar Extremes: April 27, 2009 Ten participants attended the Life in the Polar Extremes webinar presented by Jessica FriesGaither. Seven of the ten completed the online questionnaire. Overall, session content was rated by respondents as good or excellent (4.3). Reasons given for attending the session included knowledge attainment, personal interest, and presenter reputation. Respondents cited they liked the resources and ideas presented. Respondents expressed concern about too much overview of the topic and lack of direction for first-time webinar attendees. Respondents appreciated being introduced to the cited resources and activities. Additional comments included a suggestion for adding links for furthering content knowledge, a recommendation for broader coverage of the topic, and a comment expressing appreciation that the resources and content could be used immediately in the classroom.

Evaluation of Beyond Penguins and Polar Bears

Continuing Evaluation Activities Though the Beyond Penguins and Polar Bears (BPPB) project has completed its 3-year funding cycle, project staff anticipate requesting no-cost extension from the National Science Foundation (NSF) in order to continue to refine and disseminate project materials and resources during 20102011. During the next year, the E & A Center Evaluation Team will continue to provide evaluation services to the project with specific attention to gathering more data regarding the impact of the project and its materials on elementary teachers and their students. Evaluators will follow-up with Year 3 pilot teachers using questionnaires, classroom observations and interviews in order to determine how teachersâ&#x20AC;&#x2122; use of the materials and resources changes over time and the subsequent impact on the teachersâ&#x20AC;&#x2122; instructional practice. Students of these teachers also will respond to questionnaires and may participate in focus groups regarding their experiences with materials and impact on their views and understandings of science. Findings of the Year 3 evaluation suggested that the project had more impact on younger students (i.e., those in Grade 3) than on those in Grades 4 and 5. As use of the BPPB Student Questionnaire was limited to students in Grades 3 and above, we were unable to gather data on students younger than Grade 3. During the next year, the evaluation team will propose conducting classroom observations and interviews in Grades K-2 classrooms in order to learn more about the impact on these students and their teachers and the extent to which the BPPB materials and resources improve teacher and student understandings of polar science and attitudes toward science learning. Additionally, the final issue of the BPPB e-magazine will be published June 2010. Evaluators will conduct a final full-site expert review for the summative report in 2011.

Evaluation of Beyond Penguins and Polar Bears

Summary and Recommendations The following observations are offered regarding progress toward BPPB project goals: •

According to field-test teacher journal entries, quality on-line resources were accessible to the target audience. As of April 14, 2010, 19 issues of the online magazine had been published to the BPPB website. Field-test teachers reported the resources were easy to use, aligned with standards and curricula, and engaging to students. •

Webmetrics data and field-test teacher journal entiries indicated that BPPB website tools and resources were accessible to and appropriate for the target audience. Webmetric trend data indicated educator usage revolved around the academic calendar. Website usage decreased during typical academic breaks (winter and summer). Webmetric data also indicated that a great majority of visitors were finding Beyond Penguins and Polar Bears through search engines, although the majority of the search keywords used are a variation of the website name. This indicated that although the website address (URL) was still unfamiliar, the Beyond Penguins and Polar Bears name was known by visitors. •

Goal 3: Disseminating deliverables through presentations, publications, digital libraries, and push technologies.

Dissemination activities were ongoing and continued to reach members of the target audience. As of April 2010, BPPB staff members had presented at seven web seminar events and 10 conferences or meetings. BPPB promotional material was distributed at 10 educational or science events. Questionnaire data indicated that dissemination was effective at communicating information about the project and its goals. •

Goal 4: Evaluating the impact of the project deliverables on K-5 teachers and students.

Year 3 teacher pre- and post-questionnaire data showed little change in teaching practices for field-test teachers. It should be noted, however, that classroom observations and prequestionnaire data suggested that a ceiling effect might explain the lack of change in postquestionnaire data. Field-test observations and journal data also indicated a high level of reformed-teaching strategies were part of the teacher repertoires prior to exposure to project materials. Integration of science and literacy in field-test teacher classrooms was indicated extensively in the data. After exposure to project materials field-test teachers were more likely to provide opportunities to read about science and were more likely to have their students write to

Evaluation of Beyond Penguins and Polar Bears

communicate scientific results. Observation and journal data also indicated high levels of reading and writing about scientific topics. Year 3 student pre- and post-questionnaire data showed that BPPB deliverables had more effect on students of lower grades (Grade 3) than on students of higher grades (Grades 4-5). Grade 3 students reported more comfort with science after exposure to BPPB teacher resources stating that they liked science more, believed they were better at science, and understood science more. Field-test teacher data indicated an improvement in student achievement and engagement in science activities across all grade levels. General Summary and Recommendations: 1. Evidence regarding changes in teaching practices following resource implementation was not found in teacher data. Follow-up interviews, observations and questionnaire data collection are recommended to assess continuation of material usage and subsequent impact on teaching practices and on student learning. 2. Evidence regarding student impact revealed a greater impact on younger students (Grade 3) than on those in Grades 4 and 5. Although cognitive lab data suggested use of the BPPB Student Questionnaire was not recommended with students younger than Grade 3, the external evaluation team suggest collecting data from students in Grades K-2 through focus groups, interviews, and classroom observations. 3. The Beyond Penguins and Polar Bears Project Team will publish the final e-magazine issue June 2010. A final review of the entire website and materials by experts and by teacher focus groups in the target audience is recommended. 4. Evidence regarding website and resource accessibility was found in webmetrics data. Collecting, compiling and analyzing webmetric data for the entire life of the project is recommended to see overall trends of website and resource usage.

Evaluation of Beyond Penguins and Polar Bears

References Horizon Research Inc. (2000). Inside the classroom teacher interview protocol. Retrieved February 3, 2010 from: http://www.horizon-research.com/instruments/clas/interview.php Lederman, N. G. (1992). Students’ and teachers’ conceptions of the nature of science: A review of the research. Journal of Research in Science Teaching, 29(4), 331-359. Lederman, N. G., Abd-El-Khalick, F., Bell, R. L., & Schwartz, R.S. (2002). Views of nature of science questionnaire: Toward valid and meaningful assessment of learners’ conceptions of nature of science. Journal of Research in Science Teaching, 39(6), 497-521. Morell, P. D., Wainright, C., & Flick, L. (2004) Reformed teaching strategies used by student teachers. School Science and Mathematics, 104(5), 199-213. Pellegrini, A. D., & Bjorklund, D. F. (1998). Applied child study: A developmental approach. (3rd ed.). Mahwah, NJ: Earlbaum. Wainright, C., Morrell, P. D., Flick, L., & Schepige, A. (2004). Observation of reform teaching in undergraduate level mathematics and science courses. School Science and Mathematics, 104(7), 322-335.

Evaluation of Beyond Penguins and Polar Bears

Appendices Appendix A. Teacher Pre-Questionnaire ................................................................................ 38 Appendix B. Teacher Post-Questionnaire ............................................................................... 42 Appendix C. Student Questionnaire ....................................................................................... 47 Appendix D. OCEPT-Teacher Observation Protocol (O-TOP).................................................... 49 Appendix E. Inside the Classroom: Teacher Interview Protocol ............................................... 51 Appendix F. BPPB Evaluation Matrix ...................................................................................... 53 Appendix G. NSTA Web Seminar Evaluation: Arctic and Antarctic Birds .................................... 54 Appendix H. NSTA Web Seminar Evaluation: Timely Teachings: Seasons and the Cycles of Night and Day ................................................................................................................. 60 Appendix I. NSDL Web Seminar Evaluation: Brown Bag Series ................................................ 66

Evaluation of Beyond Penguins and Polar Bears

Appendix A

Beyond Penguins and Polar Bears Teacher Questionnaire Pre

PARTICIPANT INFORMATION Instructions: Please use a dark pen or pencil. Please provide answers that best represent your situation. We request the following information so that we can match this questionnaire with one you may be asked to complete in the future. Your responses will be completely confidential. No identifying information will be used in any report or paper. 1.

The first letter of my FIRST name is: Example: My first name is Chris C

Answer here:

The first letter of my LAST name is: Example: My last name is Smith S

Answer here:

My date of birth is: Example: 0 6 Month

Year

Month

Day

Year

I have been employed as a teacher for: (Please select one.)

  7.

Day

Answer here:

I am certified/licensed to teach grade(s): (Please circle all that apply.) K

8 1

During this school year, I am teaching grade(s): (Please circle all that apply.) K

0-1 year 2-5 years

 



6-10 years

Over 15 years

11-15 years

My highest degree earned is: (Please select one.)

 

Bachelor’s Masters

Evaluation of Beyond Penguins and Polar Bears

 

Masters + 30 credit hours or more Doctorate

Beyond Penguins and Polar Bears Teacher Questionnaire Pre I have a degree in a science-related discipline (e.g. chemistry, biology.)

 9.

Yes



Which best describes your professional development experience in science (e.g., workshops, online courses, interactions with science organizations) since receiving your certificate/license? (Please select one.)

  

I have participated in no science professional development. I have participated in some science professional development. I have participated in much science professional development.

10. My gender is: (Please select one.)



Female



Male

11. My background is best described as: (Please select one.)

  

African American American Indian/Alaskan Native Asian or Pacific Islander

  

Latino/a White Other

2 Evaluation of Beyond Penguins and Polar Bears

Beyond Penguins and Polar Bears Teacher Questionnaire Pre

HOW I TEACH Please circle the response that best reflects how often each practice occurs in your classroom. Please circle “Not Applicable” if the teaching practice is not appropriate for your students’ grade-level.

Almost Never Seldom Sometimes Often Very Often Not Applicable

In my classroom, I ... 1.

arrange seating to facilitate student discussion.

use open-ended questions.

require that my students supply evidence to support their claims.

encourage questions from my students.

allow my students to work at their own pace.

encourage my students to explain concepts to one another.

encourage my students to consider alternative explanations.

discuss the work of scientists.

provide time for my students to discuss subject-specific ideas among themselves.

10. discuss experiments from the history of science.

11. use non-traditional/authentic assessments.

12. give lectures.

13. integrate science instruction with literacy by providing opportunities for reading.

14. integrate science instruction with literacy by providing opportunities for writing.

15. integrate science instruction with literacy by providing opportunities for discussion.

16. choose nonfiction, science-themed articles and books for my classroom.

17. choose nonfiction articles and books that are not science-themed for my classroom.

18. regularly include science texts in my reading instruction.

19. frame science concepts using grade-appropriate real world applications. Source: Adapted from E&A, CUNY, 2008-09 Instruments Evaluation of Beyond Penguins and Polar Bears

Beyond Penguins and Polar Bears Teacher Questionnaire Pre

WHAT MY STUDENTS DO Please circle the response that best reflects how often each practice occurs in your classroom. Please circle “Not Applicable” if the teaching practice is not appropriate for your students’ grade-level. In my classroom, my students ...

Almost Never Seldom Sometimes Often Very Often Not Applicable

use data to justify responses to questions.

argue or debate with one another about the interpretation of data.

repeat experiments to confirm results.

use multiple sources of information to learn.

consider alternative explanations to accepted theories.

design activities to test their own ideas.

consult one another as sources for learning.

talk with one another to promote learning.

use educational technology in class.

10. develop scientific literacy skills.

11. learn about real world applications of science.

12. take notes and listen to lectures.

13. do worksheets.

14. learn scientific facts by using worksheets.

15. memorize scientific facts.

16. use science as the basis for authentic reading.

17. use science as the basis for authentic writing.

18. write to communicate and explain scientific results.

19. read science-themed, non-fiction in class.

Source: Adapted from E&A, CUNY, 2008-09 Instruments

4 Evaluation of Beyond Penguins and Polar Bears

Appendix B

Beyond Penguins and Polar Bears Teacher Questionnaire Post

The first letter of my FIRST name is: Example: My first name is Chris C

Answer here:

The first letter of my LAST name is: Example: My last name is Smith S

Answer here:

My date of birth is: Example: 0 6 Month

8 1

Day

Year

Evaluation of Beyond Penguins and Polar Bears

Answer here: Month

Day

Year

Beyond Penguins and Polar Bears Teacher Questionnaire Post

Almost Never Seldom Sometimes Often Very Often Not Applicable

In my classroom, I ... 1.

arrange seating to facilitate student discussion.

use open-ended questions.

require that my students supply evidence to support their claims.

encourage questions from my students.

allow my students to work at their own pace.

encourage my students to explain concepts to one another.

encourage my students to consider alternative explanations.

discuss the work of scientists.

provide time for my students to discuss subject-specific ideas among themselves.

10. discuss experiments from the history of science.

11. use non-traditional/authentic assessments.

12. give lectures.

13. integrate science instruction with literacy by providing opportunities for reading.

14. integrate science instruction with literacy by providing opportunities for writing.

15. integrate science instruction with literacy by providing opportunities for discussion.

16. choose nonfiction, science-themed articles and books for my classroom.

17. choose nonfiction articles and books that are not science-themed for my classroom.

18. regularly include science texts in my reading instruction.

19. frame science concepts using grade-appropriate real world applications. Source: Adapted from E&A, CUNY, 2008-09 Instruments

2 Evaluation of Beyond Penguins and Polar Bears

Beyond Penguins and Polar Bears Teacher Questionnaire Post

Almost Never Seldom Sometimes Often Very Often

In my classroom, my students ...

Not Applicable

use data to justify responses to questions.

argue or debate with one another about the interpretation of data.

repeat experiments to confirm results.

use multiple sources of information to learn.

consider alternative explanations to accepted theories.

design activities to test their own ideas.

consult one another as sources for learning.

talk with one another to promote learning.

use educational technology in class.

10. develop scientific literacy skills.

11. learn about real world applications of science.

12. take notes and listen to lectures.

13. do worksheets.

14. learn scientific facts by using worksheets.

15. memorize scientific facts.

16. use science as the basis for authentic reading.

17. use science as the basis for authentic writing.

18. write to communicate and explain scientific results.

19. read science-themed, non-fiction in class.

Source: Adapted from E&A, CUNY, 2008-09 Instruments Evaluation of Beyond Penguins and Polar Bears

Beyond Penguins and Polar Bears Teacher Questionnaire Post

INTEGRATION OF BPPB MATERIALS Please circle the response that best reflects your experience with the Beyond Penguins and Polar Bears Project.

As a result of the “Beyond Penguins and Polar Bears” Project, I have…

Strongly Disagree Disagree Undecided Agree Strongly Agree

incorporated the BPPB content into a specific unit of study.

2. designed activities for my students that use the objectives of the BPPB project.

changed the instructional strategies that I use in my classroom.

changed the ways I assess student understanding.

changed the ways I use educational technology.

changed the science curriculum in my classroom.

increased my own science content knowledge.

8. gained confidence in teaching science to my students.

9. learned that the Arctic and Antarctica are important geographic regions.

10. become more aware of changes taking place in the polar regions.

11. become an advocate for protection of the polar regions.

Source: Adapted from The Online Evaluation Resource Library (OERL) at http://oerl.sri.com/instruments/te/teachsurv/instr50.html

5 Evaluation of Beyond Penguins and Polar Bears

Beyond Penguins and Polar Bears Teacher Questionnaire Post

EVIDENCE OF IMPACT ON STUDENTS Please circle the response that best reflects your observations of your students. As a result of the “Beyond Penguins and Polar Bears” Project, . . .

Strongly disagree Disagree Undecided Agree Strongly agree

my students like science more.

my students believe they are good at science.

my students understand more of what goes on in science.

my students have increased their knowledge of specific science concepts through the BPPB materials.

my students have learned the BPPB content.

my students’ scientific literacy has increased.

my students’ achievement/performance in science has improved.

my students’ effort in science has improved.

9. my students’ reading skills in science have improved.

10. my students’ writing skills in science have improved.

11. my students’ discussion skills in science have improved.

Source: Adapted from E&A, Penn MSP, Student Instrument

6 Evaluation of Beyond Penguins and Polar Bears

Appendix C

Beyond Penguins and Polar Bears Student Questionnaire Fall 2009

Instructions DO NOT WRITE YOUR NAME ON THIS TEST. Please use either a blue/black pen or pencil to complete this test. When you answer the questions, please place an X in only ONE box per item. For Example:

The first letter of my FIRST name is: Example: My first name is Chris C

Answer here:

The first letter of my LAST name is: Example: My last name is Smith S

Answer here:

My date of birth is: Example: 0 6 Month

9 7

Day

Year

1st

Day

Year

2nd

3rd

4th

5th

Boy

What is your age? (please select only one) 5

Month

My gender is: (please select only one) Girl

Answer here:

What grade are you in? (please select only one) K

My background is: (please select only one) African American

Latino/a

American Indian/Alaskan Native

White

Asian or Pacific Islander

Other

1 Evaluation of Beyond Penguins and Polar Bears

BPPB Student Questionnaire

Science Views No Undecided Yes Circle the response that best reflects your views. 1.

I like science.







I am good at science.







I want to learn more science.







I understand most of what goes on in science.







Boys are better at science than girls.







Almost all people use science in their jobs.







Science is useful for solving everyday problems.







Reading is important in science.







9. Writing is important in science.







10. Discussion is important in science.







Source: PENN MSP, Instruments, Student_Quest_Science101

Evaluation of Beyond Penguins and Polar Bears

Appendix D

Evaluation of Beyond Penguins and Polar Bears

Appendix E

BPPB Spring 2009-‐2010 Teacher Observation Follow-‐Up Inside the Classroom: Teacher Interview Protocol A. Learning Goals a. I’d like to know more about the students in this class. Tell me about the ability levels of students in this class. How do they compare to students in the school as a whole? i. Are there any students with special needs in this class? ii. Are there any students for whom English is not their first language? b. Is student absenteeism or mobility a problem for you in this class? c. Please help me understand where this lesson fits in the sequence of the unit you are working on. What have the students experienced prior to today’s lesson. d. What was the specific purpose of the observed lesson? e. How do you feel about how the lesson played out? What do you think the students gained from the observed lesson? f. What was the next step for the class in the observed lesson’s unit? B. The Teacher a. How do you feel about teaching this topic? i. Do you enjoy it? ii. How well prepared do you feel to guide student learning of this content?

Adapted from Horizon Research, Inc., Inside the Classroom: Teacher Interview Protocol. Evaluation of Beyond Penguins and Polar Bears

1 51

BPPB Spring 2009-‐2010 Teacher Observation Follow-‐Up iii. What opportunities have you had to learn about this particular content area?

b. How do you feel about teaching with this pedagogy?

i. How comfortable do you feel using the instructional strategies involved in teaching this lesson?

c. How many years have you been teaching prior to this year?

i. Have you taught this lesson before?

ii. How different was the observed lesson from how you have taught it previously?

iii. Is there anything about this particular group of students that led you to plan this lesson this way?

Adapted from Horizon Research, Inc., Inside the Classroom: Teacher Interview Protocol. Evaluation of Beyond Penguins and Polar Bears

2 52

Goal 1: Provide a context to online resources by creating, identifying, selecting, and adapting learning resources from the National Science Digital Library, the Ohio Resource Center, other IPY-funded projects, and additional high-quality content providers. DATA EVALUATION ACTIVITIES PARTICIPANTS • Newly created and existing science education • Expert/user review of education resources. • K-5 science teachers and other educators resources from the ORC, NSDL, other IPY• Education content experts funded projects, and other content providers. Goal 2: Modify and build communication, production, and cyberinfrastructure tools to amplify resource discovery and access to resources, increase the ease of reuse and repurposing of content, decrease production times, and increase automated dissemination of IPY materials to various audiences. DATA EVALUATION ACTIVITIES PARTICIPANTS • Communication, production, and • Expert/user review of the tools and their application to • K-5 science teachers and other educators cyberinfrastructure tools (to be developed by the creation and use/dissemination of science • NSDL Core Integration Team the NSDL core Integration team). education resources. • Science education resources. Goal 3: Disseminate deliverables through presentations, publications, digital libraries, and push technologies. DATA EVALUATION ACTIVITIES PARTICIPANTS • Online and print versions of the polar • Expert/user review of the online and print magazine, • K-5 science teachers and other educators magazine. the monthly podcasts, and Web seminars. • School librarians/media specialists • Monthly podcast series organized around each • Tracking of the number of resources submitted to the • NSDL Core Integration Team month’s magazine topic. LOR. • Web seminars to address polar science • Tracking of presentations and submitted publications. concepts. • Resources submitted to the IPY Learning Objects Repository (LOR). • Conference presentation proposals. • Journal submissions. Goal 4: Evaluate the impact of the project deliverables on K-5 elementary teachers and students. DATA EVALUATION ACTIVITIES PARTICIPANTS • Teacher Questionnaire containing items which • Pre and post comparisons of responses to the Teacher • K-5 science teachers rate and rank instructional practices, level of Questionnaire and the Student Questionnaire. • K-5 science students integration of polar concepts and literature, • Train focus group facilitator. and observed changes in student literacy in • Content analysis of classroom observations and teacher general/polar science. interviews. • Student Questionnaire containing attitudinal • Item analysis of polar science tests. items toward science and polar concepts. • Focus group interviews to assess usefulness of dissemination efforts. • Pre and post polar science assessment tests.

IPY: BEYOND PENGUINS AND POLAR BEARS EVALUATION MATRIX 2007

Appendix F

Evaluation of Beyond Penguins and Polar Bears

Appendix G

Web Seminar Participant Feedback Results NSDL: Beyond Penguins and Polar Bears: Artic and Antarctic Birds April 21, 2009

Evaluation of Beyond Penguins and Polar Bears

Web Seminar Participant Feedback Results NSDL: Beyond Penguins and Polar Bears: Artic and Antarctic Birds April 21, 2009 This report provides the participant feedback results for Web Seminar #7 for NSDL Recommendations for future iterations of Web Seminar events are provided at the conclusion of the report. WEB SEMINAR SUMMARY....................................................................................................................................3 WEB SEMINAR TOPIC AND PARTICIPANTS INFORMATION......................................................................3 WEB SEMINAR PARTICIPANT FEEDBACK RESULTS ...................................................................................4 PRESENTER’S EVALUATION: JENNIFER FEE.................................................................................................5 PRESENTER’S EVALUATION: COLLEEN MCLINN ........................................................................................5 PRESENTER’S EVALUATION: JESSICA FRIES-GAITHER ............................................................................5 RECOMMENDATION: .............................................................................................................................................5 APPENDIX A: SCREEN SNAPSHOTS FROM THE WEB SEMINAR ...............................................................6

2 Evaluation of Beyond Penguins and Polar Bears

NSTA Web Seminar NSDL: Beyond Penguins and Polar Bears: Artic and Antarctic Birds April 21, 2009

Web Seminar Participant Feedback Results NSDL: Beyond Penguins and Polar Bears: Artic and Antarctic Birds April 21, 2009 Web Seminar Summary This Web Seminar, developed in collaboration with the National Science Digital Library, took place on Tuesday, April 21, 2009 from 6:30 p.m. to 8:00 p.m. Eastern time. In this program, presenters Jennifer Fee, Colleen McLinn, and Jessica Fries-Gaither provided an overview of resources that can be used to investigate bird behavior in the classroom. Thirty-five (35) participants were in attendance at the Web Seminar and thirty-one (31) of them completed the online feedback survey at the end of the session. This Web Seminarâ&#x20AC;&#x2122;s archive can be found at: http://learningcenter.nsta.org/products/symposia_seminars/NSDL3/Webseminar7.aspx Overall, the participants rated the Web Seminar content as valuable, interactive, and relevant. Open response comments complimented the organization of the seminar and confirmed the ability of participants to engage with the facilitator and content experts via this medium. One hundred percent (100%) of the participants would like to see more of these types of synchronous online learning experiences offered. Web Seminar Topic and Participants Information Jennifer Fee, BirdSleuth Project Leader for the Cornell Lab of Ornithology introduced ways to incorporate inquiry in teaching through the study of birds. Resources such as BirdSleuth allow for students to investigate bird behavior, patterns in migration and other aspects of bird study that align to science teaching standards. Colleen McLinn, Outreach Associate for the Macaulay Library of the Cornell Lab of Ornithology featured some of the resources from Macaulayâ&#x20AC;&#x2122;s collection of animal sounds and how these materials can be used in the classroom. McLinn discussed examples of experiments using bird sounds and bird call identification, lessons studying physical adaptations, and future plans for more educational resources related to the library. Jessica Fries-Gaither highlighted resources from the Cornell Lab of Ornithology as well as additional materials in the bird issue of Beyond Penguins and Polar Bears. This cyberzine, directed at elementary educators that are easily adaptable for older grades, provides strategies, lesson plans, multimedia, and articles that support the integration of science and literacy with a focus on the Polar Regions. Educators represented the states of Alaska, California, Florida, Illinois, Kentucky, Louisiana, Maryland, Massachusetts, Minnesota, New Hampshire, New York, North Carolina, Ohio, Pennsylvania, Texas, Virginia and Wyoming.

3 Evaluation of Beyond Penguins and Polar Bears

NSTA Web Seminar NSDL: Beyond Penguins and Polar Bears: Artic and Antarctic Birds April 21, 2009

Seminar participants received a one year subscription to one of NSTA’s SciGuides and a certificate of attendance was deposited into their My Transcript area in the NSTA Learning Center for completing the evaluation form at the end of the program. Web Seminar Participant Feedback Results Session Structure and Content: Please evaluate the session using the following scale: 5 - Excellent, 4 - Good, 3 - Average, 2 - Fair, 1 - Poor Overall, the Web Seminar was valuable to me. The content of this Web Seminar was relevant to me. The interactive nature of the Web Seminar was valuable. The time the Web Seminar was held fit my schedule.

4.58/5.0 4.48/5.0 4.74/5.0 4.55/5.0

Please describe how this content was valuable or relevant to you. ! “I work at an aquarium and we have birds on exhibit but we also have local habitats that we have programs where we talk about birds. Using and incorporating some of the activity ideas will be a good way to enhance some of our programs, even for adults.” ! “Excellent resources and ideas for student activities. One of the best ever for ideas!” ! “I teach pre-service teachers and need good resources to model for them ways to teach science that is engaging, inquiry-based, technology rich, and allows for student questions and collaborations.” ! “I teach adaptations, motion, energy, flight, force, etc. in fifth grade. I am using some of this content tomorrow as review with my students. The state test for science is next week!!!” ! “Provided great web resources for ideas on how to use birds as a teaching tool.” ! “Adaptations of birds and nonfiction children's books.” Would you like to see more Web Seminars like this one offered in the future? Yes 100% No 0% Please share your overall comments about this Web Seminar, for example, what you enjoyed the most, what you found most interesting. Tell us one thing you learned by attending this Web Seminar. ! “I enjoyed the whole seminar and will likely use at least one or two activities THIS school year (which ends in 4 1/2 weeks).” ! “Very good. I enjoyed it very much!” ! “I love the addition of diigo lists!” ! “Example of using the one syllable and four emotions and comparing to bird speak.”

4 Evaluation of Beyond Penguins and Polar Bears

NSTA Web Seminar NSDL: Beyond Penguins and Polar Bears: Artic and Antarctic Birds April 21, 2009

Presenter’s Evaluation: Jennifer Fee Please evaluate the presenter using the following scale: 5 - Excellent, 4 - Good, 3 - Average, 2 - Fair, 1 - Poor Knowledge of subject Clarity of explanations Responsiveness to questions Pace of delivery

4.84/5.0 4.77/5.0 4.77/5.0 4.77/5.0

Presenter’s Evaluation: Colleen McLinn Please evaluate the presenter using the following scale: 5 - Excellent, 4 - Good, 3 - Average, 2 - Fair, 1 - Poor Knowledge of subject Clarity of explanations Responsiveness to questions Pace of delivery

4.87/5.0 4.77/5.0 4.84/5.0 4.71/5.0

Presenter’s Evaluation: Jessica Fries-Gaither Please evaluate the presenter using the following scale: 5 - Excellent, 4 - Good, 3 - Average, 2 - Fair, 1 - Poor Knowledge of subject Clarity of explanations Responsiveness to questions Pace of delivery

4.87/5.0 4.84/5.0 4.81/5.0 4.81/5.0

Additional comments from participants: ! “Thanks to NSF for the funding for Beyond Penguins and Polar Bears. And thanks to the presenters and NSTA for an excellent session.” ! “Thanks great job!” ! “Thank you - can't imagine all the hard work these three wonderful people are putting in so that we have great resources.” ! “Great job ladies ! Nice to hear such a knowledgeable group of young scientists.” ! “Looking forward to the next one.” ! “This web seminar was easy for all levels of science teachers to not only understand the information but great ideas for future application of information.” Recommendation: ! “More time with how lessons can be modified for upper grades from teachers who have used them.”

5 Evaluation of Beyond Penguins and Polar Bears

NSTA Web Seminar NSDL: Beyond Penguins and Polar Bears: Artic and Antarctic Birds April 21, 2009

Appendix A: Screen Snapshots from the Web Seminar

Figure 1: The presenters started the audience thinking about the importance of birds.

Figure 2: The presenter asked the participants for responses.

Figure 3: Participants were introduced to classroom activities.

6 Evaluation of Beyond Penguins and Polar Bears

Appendix H

Web Seminar Participant Feedback Results NSDL: Timely Teachings: Seasons and the Cycles of Night and Day December 10, 2009

Evaluation of Beyond Penguins and Polar Bears

Web Seminar Participant Feedback Results NSDL: Timely Teachings: Seasons and Cycles of Night and Day December 10, 2009 This report provides the participant feedback results for Web Seminar that was conducted with our partners from NSDL. Recommendations on future iterations of Web Seminar events are provided at the conclusion of the report. WEB SEMINAR SUMMARY....................................................................................................................................3 WEB SEMINAR TOPIC AND PARTICIPANTS INFORMATION......................................................................3 WEB SEMINAR PARTICIPANT FEEDBACK RESULTS ...................................................................................4 PRESENTERâ&#x20AC;&#x2122;S EVALUATION: JESSICA FRIES-GAITHER ............................................................................5 RECOMMENDATION: .............................................................................................................................................5 APPENDIX A: SCREEN SNAPSHOTS FROM THE WEB SEMINAR ...............................................................6

Evaluation of Beyond Penguins and Polar Bears

December 10, 2009

Web Seminar Evaluation Results NSDL: Timely Teachings: Seasons and Cycles of Night and Day December 10, 2009 Web Seminar Summary This Web Seminar Presented by Jessica Fries-Gaither, in collaboration with the National Science Digital Library and The Ohio State University, took place on Thursday, December 10, 2009 from 6:30 p.m. to 8:00 p.m. Eastern time. Sixty-six (66) participants were in attendance at the Web Seminar and fifty-three (53) of them completed the online feedback survey at the end of the session. This Web Seminarâ&#x20AC;&#x2122;s archive can be found at: https://sas.elluminate.com/site/external/recording/playback/link/meeting.jnlp?suid=M.429240994BB06F86958BB11BEC7EDD

Overall, the participants rated the Web Seminar content as valuable, interactive, and relevant. Open response comments complimented the organization of the seminar and confirmed the ability of participants to engage with the facilitator and content experts via this medium. Ninetyeight percent (98%) of the participants would like to see more of these types of synchronous online learning experiences offered.

Web Seminar Topic and Participants Information Jessica Fries-Gaither, Project Director for Beyond Penguins and Polar Bears and Education Resource Specialist at The Ohio State University started the presentation with a content review relating to the seasons. Some concepts related to seasons are hard for students to understand and lead to misconceptions Educators represented the states of California, Colorado, Florida, Georgia, Illinois, Indiana, Iowa, Louisiana, Maryland, Massachusetts, Minnesota, Missouri, Montana, New Hampshire, New Jersey, New York, North Carolina, Ohio, Oklahoma, Pennsylvania, Tennessee, Texas, Vermont, Virginia and Wisconsin. Three educators joined the program from countries outside of the United States: Canada, Spain, and Romania. Seminar participants received a one year subscription to one of NSTAâ&#x20AC;&#x2122;s SciGuides and a certificate of attendance was deposited into their Learning Center My Transcript for completing the evaluation form at the end of the program.

Evaluation of Beyond Penguins and Polar Bears

December 10, 2009

Web Seminar Participant Feedback Results Session Structure and Content: Please evaluate the session using the following scale: 5 - Excellent, 4 - Good, 3 - Average, 2 - Fair, 1 - Poor Overall, the Web Seminar was valuable to me. The content of this Web Seminar was relevant to me. The interactive nature of the Web Seminar was valuable. The time the Web Seminar was held fit my schedule.

436/5.0 4.38/5.0 4.51/5.0 4.26/5.0

Please describe how this content was valuable or relevant to you. • • • • • •

I am aware of many of the misconceptions students have today that were covered. It is always good to see where others are at with these concepts and the man resources presented in the Webinar will be useful. I found the discussion informative; I particularly liked the format of looking at student misconceptions as an entry point. I appreciate resource information that Jessica and participants shared as well. I am beginning lessons after break on seasons and the solar system and this was great refresher course. I teach this as part of my astronomy unit - Earth/sun/moon interactions. The topic is something that I am teaching to 5th graders so will be helpful with the visuals. Great resources for correcting student misconceptions on space science.

Would you like to see more Web Seminars like this one offered in the future? Yes – 98% No - 2% Please share your overall comments about this Web Seminar, for example, what you enjoyed the most, what you found most interesting. Tell us one thing you learned by attending this Web Seminar. • • • • • •

Because so many of my students that struggle with this, are international, there are some strategies that will be very useful. Lots of good connections for those teaching younger students This was my first time but definitely won't be my last! I am interested in exploring the resources and I joined the middle school science portal!! Lots of links and resources. I'll check them and get the books. My son will love it. I will also share it with other parents. I work with parents of gifted children. Excellent presentation and resources

Evaluation of Beyond Penguins and Polar Bears

December 10, 2009

4.83/5.0 4.72/5.0 4.79/5.0 4.72/5.0

Additional comments from participants: • • • • • •

Great job Jessica had a wonderful "on-line" personality that came through very well. Some people are flat when you aren't seeing them in person, she was definitely 3-D:) Thanks for doing this seminar. I'll recommend it to my school too. Thank you so much! Wonderful!! I miss the horse shoe. Did my bachelors in Zoology and Masters in Exercise Physiology at OSU. I miss it so much. Great webinar with good resources.

Recommendation: • •

Go into further detail about how to explain some of the more abstract concepts to students. Middle school students respond more to interactive representations. Resources like that would be helpful.

Evaluation of Beyond Penguins and Polar Bears

NSTA Web Seminar NSDL: Timely Teachings: Seasons and Cycles of Night and Day December 10, 2009

Appendix A: Screen Snapshots from the Web Seminar

Figure 1: The presenter polled the participants with questions about the seasons.

Figure 2: The presenter asked for audience feedback dealing with misconceptions.

Figure 3: Participants responded to the use of childrenâ&#x20AC;&#x2122;s literature in their class.

6 Evaluation of Beyond Penguins and Polar Bears

Appendix I

NSDL Brown Bag Report Date of Report: 4.27.09 Brown Bag Title: Beyond Penguins and Polar Bears: Integrating Science and Literacy in the Elementary Classroom---Life in the Polar Extremes Date of Program: 4.7.09, 6:30pm-7:45pm Eastern Moderator: Robert Payo Presenter: Jessica Fries-Gaither, Project Director Beyond Penguins and Polar Bears, Ohio State University This webinar was developed in collaboration with Ohio State University and Stafford County Public Schools, VA. Number of Participants: 10 (8 stayed for the duration with 2 staying for 30 minutes) Number of Registrants: 23 Number of completed surveys: 7 Participants learned about the seminar through Rita Truelove, science coordinator for Stafford County Public Schools, NSDL channels and Beyond Penguins. Brown Bag Overview Jessica Fries-Gaither provided an overview of resources related to Beyond Penguins and Polar Bears with a focus on science concepts, misconceptions and literacy strategies related to teaching ecosystems and life in extreme climates. Survey Feedback Results Rate the quality of session content (1-poor to 5-excellent)

4.3 avg

Participant comments What was the reason you attended this session? • Currently teaching ecosystems • Personal interest • The presenter • I'm teaching habitats and wanted some tips or resources to use. • Support my teachers. What did you like most about the session? • The online resources • The wonderful ideas that were given as well as the resources! • Good presentation, trusted content - good 'tips for use'

Evaluation of Beyond Penguins and Polar Bears

What did you like least about the session? • The topic overview a little too much. • I wasn't sure of how this thing worked. I didn't know if I was supposed to take notes and write everything down...is there a list of all these resources somewhere? I really didn't know what I was getting into and I feel like a lot of teachers did not sign up for this because no one really understood what this was about or what you had to do while listening. [Note: This was from a participant outside of Stafford County teachers] What was the biggest takeaway for you from this session? • Resources • The web resources and the fact that I can use it tomorrow • The diigo list Additional comments • I hope we can continue this relationship next year as well. I think we have really reached some folks. I had some great comments from some middle school folks. • These brown bags are a great add-on for the site - as you repurpose things, consider offering the highlights distributed at conferences as pdfs for downloading/archiving. what next? since you are versed in climate change... move from the poles and look at what polar science predicts for temperate/tropical regions, rainforests, etc with the same 'eye' to education? • I liked [the] presentation, but I guess it would be neat to have a broader topic. We don't even "need" to teach the tundra in 3rd grade anymore (I still do) but, for me, I think if I were going to do this again, I would want it on an entire SOL rather than just a tiny aspect/detail of it. Or if it wasn't on an entire topic, then at least give the presentation on the detail and give us more resources on the broader topic. • I enjoy all presentations given by Robert and Jessica! Very professional and well organized! The BEST part is that I can go to school tomorrow and use things I learned about tonight! YIPEE!! • I would participate in another if Ms Truelove is able get NSDL to do another one or two especially if I could see something on weather and plants. • I really enjoyed this. Thank you.

Evaluation of Beyond Penguins and Polar Bears