MSP2 Evaluation Report 2009-2010 by Digital Libraries @ Ohio State University

Evaluation of Middle School Portal 2: Math & Science Pathways (MSP2) Annual Report 2009-2010

Miami University 408 McGuffey Hall Oxford, OH 45056

Phone: 513-529-1686 Fax: 513-529-2110 Website: http://ohioeval.muohio.edu

Please cite as follows: Woodruff, S. B., Morio, K. L., & Li, Y. (2010). Evaluation of middle school portal 2: Math & science pathways (MSP2) Annual report 2009-2010. Oxford, OH: Miami University, Ohio’s Evaluation & Assessment Center for Mathematics and Science Education. Distributed by Ohio’s Evaluation & Assessment Center for Mathematics and Science Education Sarah B. Woodruff, Director 408 McGuffey Hall Miami University Oxford, Ohio 45056

Evaluation of MSP2

Middle School Portal 2: Math & Science Pathways (MSP2) Ohioâ&#x20AC;&#x2122;s Evaluation & Assessment Center for Mathematics and Science Education

Miami University Oxford, Ohio

Sarah B. Woodruff Principal Investigator Kristen Morio Project Director Yue Li Senior Statistician

Evaluation of MSP2

iii

Table of Contents Table of Contents ................................................................................................................. iv List of Tables .......................................................................................................................vii List of Figures ..................................................................................................................... viii Introduction .......................................................................................................................... 1 Project Description ............................................................................................................ 1 Evaluation............................................................................................................................. 2 Participants ....................................................................................................................... 2 Instruments ...................................................................................................................... 2 Profile Questions Survey ................................................................................................ 2 MSP2 Website Questionnaire .......................................................................................... 3 MSP2 Educator Questionnaire......................................................................................... 3 MSP2 Youth Questionnaire ............................................................................................. 3 MSP2 Teacher Leader Interview Protocol......................................................................... 3 Web-based Resource Review Rubric................................................................................ 3 MSP2 Participation Rank Rubric ...................................................................................... 3 MSP2 Social Network Conversation Rubric ....................................................................... 3 Data Collection .................................................................................................................. 4 Data Analysis .................................................................................................................... 4 Findings................................................................................................................................ 6 Goals and Timeline ............................................................................................................ 6 Profile Questions Survey .................................................................................................. 10 MSP2 Website Questionnaire ............................................................................................ 12 MSP2 Educator Questionnaire........................................................................................... 14 MSP2 Youth Questionnaire ............................................................................................... 15 Webmetrics..................................................................................................................... 15 MSP2 Ning Social Networking Website........................................................................... 16 New and Returning Visitors Comparison .................................................................... 16 Number of Visits .................................................................................................. 16 Average Page Views per Visit................................................................................ 17 Average Time Spent on MSP2 Ning ....................................................................... 18 MSP2 Wiki Pages ......................................................................................................... 21 Teacher Leader Interviews ............................................................................................... 22

Evaluation of MSP2

Teacher Leader Demographics...................................................................................... 22 Teacher Leader Responsibilities .................................................................................... 22 Comparisons of Social Networks ................................................................................... 23 Participation ................................................................................................................ 23 MSP2 Ning Features..................................................................................................... 23 Improvements............................................................................................................. 24 Face-to-Face vs. Web-based Professional Development .................................................. 24 Member Problems or Concerns ..................................................................................... 24 Document Review............................................................................................................ 25 Resource Guides.......................................................................................................... 25 Blogs .......................................................................................................................... 26 Thematic Website Review ................................................................................................ 27 Dissemination.................................................................................................................. 29 Presentations at Local and National Conferences and Meetings ....................................... 29 Promotional Materials Distributed at Conferences and Meetings ...................................... 29 Webinars .................................................................................................................... 30 Webinar 1: Interactive Notebooks............................................................................. 30 Webinar 2: Moodle for Middle School ........................................................................ 30 Webinar 3: Digital Storytelling .................................................................................. 30 Webinar 5: Interactive Whiteboards .......................................................................... 31 Webinar 7: Diigo: Social Bookmarking ....................................................................... 31 Webinar 8: UDL PDQ ............................................................................................... 31 Webinar 9: Laboratory Safety ................................................................................... 32 Webinar 14: Digital Tools and Math .......................................................................... 32 Tapped - In Events ...................................................................................................... 32 External Advisory Board Meeting................................................................................... 33 Continuing Evaluation Activities ............................................................................................ 34 Summary and Recommendations.......................................................................................... 35 References.......................................................................................................................... 37 Appendices ......................................................................................................................... 38 Appendix A: Profile Questions Survey .................................................................................... 39 Appendix B: MSP2 Website Questionnaire ............................................................................. 40 Appendix C: MSP2 Educator Questionnaire ............................................................................ 45 Appendix D: MSP2 Youth Questionnaire ................................................................................ 48 Appendix E: MSP2 Teacher Leader Interview Protocol ............................................................ 58 Appendix F: Web-Based Resource Review Rubric ................................................................... 60

Evaluation of MSP2

Appendix G: MSP2 Participation Rank Rubric ......................................................................... 61 Appendix H: MSP2 Social Network Conversation Rubric ......................................................... 62 Appendix I: MSP2 Educator Questionnaire Analysis ................................................................ 63 Appendix J: MSP2 Youth Questionnaire Analysis .................................................................... 74 Appendix K: MSP2 Web Resource Score Sheet: Resource Guides ............................................ 91 Appendix L: MSP2 Web Resource Score Sheet: Blogs ............................................................. 98 Appendix M: Webinar Analyses ............................................................................................105

Evaluation of MSP2

List of Tables Table 1. Year 2 Proposed Tasks and Progress Crosswalk ............................................................ 6 Table 2. Profile Questions Survey: Respondent Gender, Spring 2010 ........................................... 10 Table 3. Profile Questions Survey: Respondent Position or Title, Spring 2010............................... 11 Table 4. Profile Questions Survey: Respondent Years of Teaching Experience, Spring 2010 .......... 11 Table 5. Profile Questions Survey: Respondent Grade Level Assignment, Spring 2010 .................. 11 Table 6. Profile Questions Survey: Respondent Subject Area, Spring 2010 ................................... 12 Table 7. MSP2 Website Questionnaire: Respondent Gender by Participation Rank ........................ 12 Table 8. MSP2 Website Questionnaire: Respondent Title by Participation Rank............................. 13 Table 9. MSP2 Website Questionnaire: Respondent Teaching Experience by Participation Rank....................................................................................................................................... 13 Table 10. MSP2 Website Questionnaire: Respondent Subject Area by Participation Rank............... 14 Table 11. MSP2 Ning Visits, May 2009 - January 2010 ................................................................ 16 Table 12. Connecting News to the National Science Education Standards Blog Visits, May 2009 -January 2010.......................................................................................................... 20 Table 13. Exemplary Resources for Middle School Math and Science Blog, May 2009 January 2010 .......................................................................................................................... 21 Table 14. MSP2 Wiki Page Visits, May 2009 - January 2010 ........................................................ 22 Table 15. Resource Review Average Scores for Science and Mathematics MSP2 Resource Guides, Spring 2010 ................................................................................................................ 25 Table 16. Resource Review Average Scores for Science and Mathematics Blog Entries, Spring 2010 ............................................................................................................................ 26 Table 17. Discussion Forum Topics and Replies, August 2009 and March 2010............................. 28 Table 18. MSP2 Blog Activity, August 2009 and November 2009. ................................................ 28 Table 19. Diigo Group Members, Bookmarks and Visits, August 2009 and February 2010.............. 29

Evaluation of MSP2

vii

List of Figures Figure 1. Comparison of number of visits of new and returning visitors to MSP2 Ning, May 2009 – April 2010. ................................................................................................................... 17 Figure 2. Comparison of average page views per visit of new and returning visitors to MSP2 Ning, May 2009 – April 2010. ................................................................................................... 18 Figure 3. Comparison of average time spent by new and returning visitors on MSP2 Ning, May 2009 – April 2010. ............................................................................................................ 19

Evaluation of MSP2

viii

Introduction Ohio’s Evaluation & Assessment Center for Mathematics and Science Education (E & A Center) is the external evaluator for the Middle School Portal 2: Math & Science Pathways (MSP2) Project. Dr. Sarah B. Woodruff, Miami University, is the Principal Investigator for the evaluation, and Kristen Morio, Research Associate, is the Project Director. Yue Li is the Senior Researcher and Statistician for the project. Dr. Woodruff and Mrs. Morio oversee all aspects of the evaluation. This report is divided into four sections. Section I provides background information about the MSP2 project and its goals. Section II provides information about Year 2 evaluation activities. Section III provides information on future evaluation plans and ongoing evaluation activities. Section IV summarizes the Year 2 evaluation and provides recommendations to the Project Team.

Project Description The Middle School Portal 2: Math & Science Pathways (MSP2) Project is a collaboration among Ohio State University College of Education (OSU), National Middle School Association (NMSA), and the Education Development Center (EDC). The Middle School Portal 2: Math & Science Pathways project (MSP2) supports middle grades educators and youth with powerful connections to people and high-quality resources, and through knowledge sharing and creation within and beyond the MSP2 community. The project also will add a youth component to the website1. The five goals of the project are to: •

Goal 1: Select, organize, develop, and contextualize quality learning resources in the areas of science and mathematics, 21st century skills, STEM careers, and middle-school education.

•

Goal 2: Design a portal that provides access to content, interactive opportunities, and virtual support for educators and youth, and facilitates the creation of a virtual professional learning community (VPLC) of middle school mathematics and science educators.

•

Goal 3: Promote resource discovery and usage through search engine optimization, training and workshops, and dissemination.

•

Goal 4: Collaborate with multiple organizations to sustain project resources and continue to meet the needs of the middle school mathematics and science community after funding ends.

•

Goal 5: Evaluate the impact of project deliverables on educators and youth, and determine how developers of digital resources and collections can best support the educational use of Web 2.0 tools and services.

The youth component or Virtual Learning Experiences (VLE) will be added to the MSP2 Ning in Year 3.

Evaluation of MSP2

Evaluation Overarching evaluation efforts focus on monitoring project implementation and assessing progress toward project goals. During the Year 1 evaluation, E & A Center staff worked with the Project Team on the development and refinement of the evaluation matrix. Year 1 evaluation activities included monitoring communication and dissemination methods (e.g., wiki activity, meetings, webinars), collecting and analyzing website statistics (webmetrics), and tracking progress on proposed project tasks. One focus of the Year 2 evaluation was to identify a profile of participation for users. Along with continuation of Year 1 evaluation activities, the Year 2 evaluation included quantitative and qualitative analyses of user participation in the Middle School Portal 2 (MSP2) Ning social network. Teacher Leader interviews and member surveys provided data for these analyses. Expert reviews of learning resources available via the MSP2 portal also were conducted.

Participants The Middle School Portal 2: Math & Science Pathways (MSP2) project focuses on mathematics, science, and technology for middle school educators. Therefore, individuals involved with middle school education (teachers, administrators, teacher educators) who joined the MSP2 Ning social network to take part in project activities were participants in the Year 2 evaluation. Teacher Leaders (TL) were recruited by the Project Team to serve as model participants in the social network and to recruit and encourage member participation in the MSP2 Ning. The first TL cohort consisted of four middle school educators. They were recruited in March 2009. Two TLs were recruited September 2009 and one more in February 2010 to make up Cohort 2. TLs were interviewed in March 2010. The Education Development Center (EDC) collected data from educators and students of Grades 5-8. These data were collected with two online instruments. Respondents were recruited through partner websites (i.e., NMSA website, MSP2 Ning) as well as on their own networks, such as the ITEST Learning Resource Center (http://itestlrc.edc.org/). The EDC also collected data from 10 students, between the ages of 11 and 14, designated as the MSP2 Virtual Learning Experience (VLE) Design Team. The EDC conducted several Design Team focus groups. Data from these focus groups were used to develop the VLE template design.

Instruments Year 2 evaluation data were collected from seven sources: (a) a member profile survey, (b) an online MSP2 member questionnaire, (c) Teacher Leader (TL) interviews, (d) resource reviews, (e) participation assessments, (f) an Education Development Center (EDC) questionnaire administered to educators, and (g) an EDC questionnaire administered to youth.

Profile Questions Survey A Profile Questions Survey was created by the Project Team to collect data on website visitors who signed up for membership to the MSP2 Ning community. The online survey was composed of six multiple-choice items and one open-response item. The Profile Questions Survey can be found in Appendix A.

Evaluation of MSP2

MSP2 Website Questionnaire The MSP2 Website Questionnaire was an online questionnaire developed by the Evaluation Team It consisted of five open-response items collecting data on respondents’ MSP2 Ning experiences. A link to the online questionnaire was emailed to all Ning members on December 10, 2009. The MSP2 Website Questionnaire can be found in Appendix B.

MSP2 Educator Questionnaire The MSP2 Educator Questionnaire was an online questionnaire developed by the Project Team to collect data on the use of technology by educators and their students. Data collected from this questionnaire, from the MSP2 Youth Questionnaire, and from Design Team focus groups will be used to develop the youth component of the project. The questionnaire consisted of nine openresponse items, three multiple-choice items, and one multi-part item rated on a 4-point rating scale ranging from strongly agree (1) to strongly disagree (4). This rating scale had no neutral or undecided response choice. The MSP2 Educator Questionnaire can be found in Appendix C.

MSP2 Youth Questionnaire The MSP2 Youth Questionnaire was an online questionnaire developed by the Project Team to collect data on personal and educational use of technology by middle school students. Data collected from this questionnaire, from the MSP2 Educator Questionnaire, and from Design Team focus groups will be used to develop the youth component of the project. The questionnaire consisted of 41 items. Eight Likert-type items were rated on a 5-point scale ranging from strongly agree (1) to strongly disagree (5). The remaining items included 17 multiple-choice items and 16 open-response items. The MSP2 Youth Questionnaire can be found in Appendix D.

MSP2 Teacher Leader Interview Protocol The MSP2 Teacher Leader Interview Protocol was created by the Evaluation Team and consisted of 14 items. The protocol collected data on Teacher Leaders’ experiences with the MSP2 Ning and social networking in general. The MSP2 Teacher Leader Interview Protocol can be found in Appendix E.

Web-based Resource Review Rubric The Web-based Resource Review Rubric was created by the Evaluation Team and was a modified version of the Ohio Resource Center ORC Resource Review Rubric (Ohio Resource Center, n.d.). Using the rubric, MSP2 Ning resource materials were rated on characteristics such as relevance, adaptability, accuracy, and appeal. The Web-based Resource Review Rubric can be found in Appendix F.

MSP2 Participation Rank Rubric The MSP2 Participation Rank Rubric was created by the Evaluation Team to assist in the development of participation profiles for various levels of use of the MSP2 Ning. The rubric consisted of ratings ranging from no participation (0) to extensive user (4) and assessed activities such as “Personalization,” “Blogs/Discussions,” “Group Membership,” “Comments,” “Events,” and discussion initiation activities. The MSP2 Participation Rank Rubric can be found in Appendix G.

MSP2 Social Network Conversation Rubric The MSP2 Social Network Conversation Rubric was created by the Evaluation Team and was a modified version of an E & A Center rubric used to evaluate online learning communities. Using the rubric, MSP2 Ning group discussions, forum discussions, and blog posts were evaluated for conversation quality based upon number of responses, number of participants, and relevance,

Evaluation of MSP2

depth, and thoughtfulness of responses. The MSP2 Social Network Conversation Rubric can be found in Appendix H.

Data Collection Data were collected from MSP2 Ning members. When users selected the â&#x20AC;&#x153;Sign Upâ&#x20AC;? link on the main page of the MSP2 Ning website (http://www.msteacher2.org), they were directed to create a username and password. New members then were asked to complete the Profile Questions Survey.2 Data collected from this survey were compiled by the Project Team and analyzed by the Evaluation Team. The MSP2 Website Questionnaire was developed as an online questionnaire. The link to the questionnaire was distributed to a list of MSP2 member email addresses. Data were downloaded for analyses on March 9, 2010. An Evaluation Team member conducted MSP2 Teacher Leader interviews by phone in March 2010. Interviews were transcribed for content analysis. EDC administered two online questionnaires simultaneously (MSP2 Educator Questionnaire and MSP2 Youth Questionnaire) to middle school educators and students between April 30 and June 15, 2009. Data were collected by the EDC and reported to the Evaluation Team. MSP2 web-based resources were reviewed by middle school education experts in the fields of mathematics and science. New MSP2 Resource Guides3 and blog posts (developed after September 2008) were reviewed using the Web-based Resource Review Rubric. Quantitative and qualitative data were collected. The second MSP2 external advisory board meeting was held online via the Elluminate Live! interface on July 15, 2009. An Evaluation Team member observed the meeting.

Website statistics (webmetrics) were collected by the Project Team through Google Analytics5 between May 2009 and April 2010. The Project Team also provided dissemination lists and webinar data to the Evaluation Team.

Data Analysis Data were analyzed in order to develop profiles of user participation. Instruments collected demographic data as well as data describing experiences with digital tools (i.e., MSP2 Ning, other social networks, blogs, wikis). Using demographic data from the Profile Questions Survey, respondents to the MSP2 Website Questionnaire were rated on their level of MSP2 Ning participation. An Evaluation Team member used the MSP2 Participation Rank Rubric, designed specifically for this task, to give members a participation rating between no participation (0) and extensive user (4). Participation rank (PR) scores were used to categorize MSP2 Ning members and make comparisons between levels of participation and attitudes toward digital tools. Qualitative analysis software, NVivo, was used to perform theory-driven content analysis on interview transcripts and open-response items from the MSP2 Website Questionnaire. Thematic website reviews were performed monthly from August 2009 to March 2010 to collect data on quality and quantity of participation. Number of groups, topics and responses within groups, forums, topics and responses within forums, and Diigo group members, bookmarks, and 2

Completing the Profile Questions Survey is optional and does not effect membership. MSP2 Resource Guides were originally developed for the NSDL Middle School Portal project and were referred to as Explore in Depth publications for that project and for the Year 1 MSP2 Evaluation Report. 4 Elluminate Live! is a web conferencing program that includes video, audio, and participation tools for a fully interactive web-based seminar environment. 5 Google Analytics is a web-based tool through which website statistics can be collected and analyzed. 3

Evaluation of MSP2

visits were counted. Group and forum discussions were randomly selected and assessed for quality of conversation using the MSP2 Social Network Conversation Rubric. Web-based resource reviews were conducted in February 2010. Two middle-school educators reviewed MSP2 web-based resources using the Web-based Resource Review Rubric created by the Evaluation Team specifically for this project. All new MSP2 Resource Guides and at least 10% of new posts for each blog (Exemplary Resources for Middle School Math and Science and Connecting News with National Science Education Standards) were reviewed.6 Blog posts were randomly selected for review. Blog posts and Resource Guides were separated and reviewed independently for mathematics and science. Webmetrics were analyzed for trends in website usage.

Resource items dated September 2008 or later are considered new for this project.

Evaluation of MSP2

Findings Quantitative and qualitative data were used to evaluate the progress of the project toward project goals. The Goals and Timeline Crosswalk displayed progress on proposed tasks for Year 2. A profile of participation was created using the Profile Questions Survey and the MSP2 Website Questionnaire. Educational Development Center (EDC) data collected from the MSP2 Educator Questionnaire and MSP2 Youth Questionnaire were used to analyze student and educator use of web tools and will be added to focus group data collected for the development of the youth component (Virtual Learning Experiences [VLE]). Webmetrics data, Teacher Leader interviews, and thematic website reviews showed educator use of the MSP2 Ning and its components. Document reviews were conducted to analyze quality of learning resources and dissemination activities were listed and evaluated for impact.

Goals and Timeline Table 1 shows proposed tasks to be completed in Year 2 and progress on each task in relation to each project goal as reported by the Project Team. Table 1. Year 2 Proposed Tasks and Progress Crosswalk Goal

Goal 1 - Select, organize, develop, contextualize quality learning resources in sci, math, 21st cent. skills, STEM careers, and ml education

Evaluation of MSP2

Proposed Year 2 Tasks Update remaining math/science EIDs (now referred to as MSP2 Resource Guides) Write six new Resource Guides

Create â&#x20AC;&#x153;on-demandâ&#x20AC;? resource lists

Progress All existing EIDs were updated and cataloged in the NSDL cataloging system. 10 new Resource Guides were created/launched (three in Math; seven in Science). Resource lists are continuously generated through MSP2 blogs (51 new blog posts; Webinar Archive page; Student and Teacher Opportunities pages; Tech Tool page).

Goal

Proposed Year 2 Tasks Expand MSP2 collection

Goal 1 - Select, organize, develop, contextualize quality learning resources in sci, math, 21st cent. skills, STEM careers, and ml education

Create/refine tutorials/promotional materials/informational tools based on feedback Refine MSP1 and MSP2 based on feedback

Goal 2 - Design portal that provides access to content, interaction, virtual support for educators and students, facilitate VPLs

Evaluation of MSP2

Progress Blog posts (37 for Expert Voices Middle School Math and Science; 14 for Expert Voices Connecting News and National Science Standards). The metadata from 2,404 MSP2 catalog records was transferred from OSU to the NSDL Cataloging Database (NCS) in August 2009. Since that time, an additional 747 records have been added and 539 deaccessioned. The NCS Search Service was integrated into the MSP2 social network so the collection can be searched. Bookmark; Oceans Resource Guide (print and online versions; totes to be used as an incentive for membership). NSDL Middle School Portal (MSP1) – site was updated to look like the MSP2 social network. Each page includes a link to “Connect to Colleagues” that takes MSP1 users to MSP2; new content from blogs are added via RSS; new wikibased resource guide added as available. MSP2 Ning continues to be updated–added: a feature that allows users to search the MSP2 collection by keyword and also to conduct canned searches; a polling feature; RSS feeds from external blogs; Groups continue to grow as well as membership.

Goal

Proposed Year 2 Tasks Continue use of networking and collaboration sites (e.g., Curriki, Tapped In, FaceBook)

Recruit/train Cohort 2 of Teacher Leaders

Goal 2 - Design portal that provides access to content, interaction, virtual support for educators and students, facilitate VPLs

Cohort 1 Teacher Leaders (TL) facilitates/recruits/presents/publishes

Digital story workshop at NMSA meeting Continue to design youth VLEs

Evaluation of MSP2

Progress MSP2 content continues to be added to the Curriki website Twitter account (54 members); Diigo groups active; continue to have TappedIn monthly chats and content continues to be added; content is being added to Slideshare, Issuu.com, bit.ly, and LearnCentral. Two new Teacher Leaders joined in Summer, 2009 Four other teachers who had been recommended by colleagues were invited. They subsequently declined Invitation/announcement about TL were framed as professional development opportunities. One applicant joined. Four TL have been directly involved in leading three webinars (Sept 2009March 2010); three TLs presented sessions at the NMSA Annual Conference; Continue to recruit welcome and mentor members of the MSP2 Ning. Content topic of summer 2010 workshop to be held at OSU. Youth and educator surveys designed and administered (445 youth; 631 educators). Design Team recruited and utilized. VLE template designed and finalizing Drupal implementation.

Goal

Proposed Year 2 Tasks Develop new face-to-face workshops Pilot MSP2 with OSU middle school preservice cohort Continue to present and exhibit 4 webinars, monthly Tapped In, Second Life events

Goal 3 - Promote resource discovery and usage through seo, trainings, dissemination

Continue to publish and share information

Implement a series of Year 2 online ‘launch’ activities associated with new EIDs, new VLEs, and new MSP2 site features.

Continue research of sponsorship opportunities, talk with potential sponsors

Goal 4 - Collaborate for sustainability

Evaluation of MSP2

Progress Continue to develop these – will occur during Summer 2010. A number of OSU preservice teachers became MSP2 members. Two webinars in July, three in August, two in September, three in October, one in November, two in February, two in March, three in April; Second Life events will occur in Year 3. Two Journal Articles in Middle Ground Journal; ads in NMSA conference book; ads in Middle School Journal. Broadcasts, blast emails, middle-E connections announcements; announcements on NMSA home page; OSU EHE biweekly newsletter; EDC listservs and project networks; MSP2 biweekly newsletter; Twitter tweets. MSP2 is a finalist along with Curriki in the MacArthur/HASTAC 2010 competition. Exploring various NSF proposals solicitations, while considering foundation grants (e.g., Pepsico Foundation, Xprize foundation). Held conversations with organizations interested in partnering with MSP2 on the current NSDL RFP, including the NSF ITEST Learning Resource Center, the Southern Regional Education Board, and EDC’s Center for Children and Technology.

Goal

Goal 4 - Collaborate for sustainability

Proposed Year 2 Tasks Pilot/evaluate e-commerce opportunities (surveys, interviews, focus groups)7

Progress MSP2 publications will be able to ‘print on demand’ through MagCloud.

Continue semi-annual advisory board Web conferences

Advisory board meeting July 2009; next meeting – June 3, 2010; continue to send updates. Launched “Help Us Help You!” poll; TL interviews; Webmetrics continue; Online Survey.

Continue evaluation efforts (online surveys, focus groups, interviews, web metrics, uptake of content across locations) of deliverables and applications of Web 2.0 tools Share preliminary information

Goal 5 - Evaluate impact

Profile Questions Survey As of March 31, 2010, 835 MSP2 Ning members responded to the Profile Questions Survey developed by the Project Team. A majority of respondents were female (72%, Table 2). Respondents were asked to provide their age and were, on average, 44 years old, with the youngest respondent indicating 15 years old and the oldest indicating 75 years old (data not displayed). Table 2. Profile Questions Survey: Respondent Gender, Spring 2010 Gender

Female Male Total

576 227 803

72 28 100

A majority of Profile Questions Survey respondents (59%) indicated that they were classroom teachers (Table 3).

Because of unforeseen organizational and financial events, NMSA had to postpone all activities related to developing an e-commerce model during FY 2009-2010. Key personnel at NMSA are currently (Spring 2010) engaged in research to develop e-commerce plan.

Evaluation of MSP2

Table 3. Profile Questions Survey: Respondent Position or Title, Spring 2010 Position or Title Classroom Teacher (preK-12) IT Support Person/Technology Integration Specialist College Faculty Student Pre-service Teacher Curriculum Coordinator Researcher Principal/Assistant Principal Informal/Home School Educator Librarian/Media Specialist Agency/Association Professional Other Total

n 480

% 59

43 33 37 21 20 14 13 9 7 96 820

5 4 5 3 3 2 2 1 1 12 100

Most respondents (40%) had more than 15 years of teaching experience (Table 4). Table 4. Profile Questions Survey: Respondent Years of Teaching Experience, Spring 2010 Years of Teaching Experience 0-1 2-5 6-10 11-15 15+ Total

18 92 105 90 200 505

3 18 21 18 40 100

Tables 5 and 6 display information about respondentsâ&#x20AC;&#x2122; classrooms. A majority of respondents (79%) indicated that they taught Grades 6-8. Most respondents (78%) either taught science or mathematics only, or in combination with another subject (66%). Table 5. Profile Questions Survey: Respondent Grade Level Assignment, Spring 2010 Grade

Preschool

Primary (K-2)

Elementary (3-5)

Middle (6-8)

484

High (9-12) Total

44 616

6 100

Note. Respondents could choose more than one option.

Evaluation of MSP2

Table 6. Profile Questions Survey: Respondent Subject Area, Spring 2010

Discipline Science Mathematics Technology English Language Arts Social Studies Foreign Language Enrichment Special Education Music & Physical Education Other Total

Taught Only this Subject 206 119 14 7 0 1 2 3

Taught Subject in Combination with Other Subjects 370 274 64 56 36 5 13 15

78 430

133 973

Note. Respondents could choose more than one option.

MSP2 Website Questionnaire MSP2 Ning members who responded to the MSP2 Website Questionnaire (n = 59) were categorized using a rubric created by the E & A Center Evaluation Team. Members were given a participation rank (PR) based on their level of activity and participation on the Ning, from no participation (0) to extensive participation (4). MSP2 Website Questionnaire data and Profile Questions Survey data then were analyzed to make comparisons between members at various participation levels. As shown in Table 7, females comprised groups with higher levels of participation. Table 7. MSP2 Website Questionnaire: Respondent Gender by Participation Rank Participant Rank 0 1 2 3 or 4

Male

Female

17 14 17 11

% 47 29 12 9

% 53 71 88 91

A majority of respondents (58%) identified themselves as PreK-12 classroom teachers. Ninetyone percent of classroom teacher respondentâ&#x20AC;&#x2122;s indicated that they taught Grades 6-8. Table 8 displays respondentsâ&#x20AC;&#x2122; position or title for each PR group. Classroom teachers tended to participate at higher levels than did other respondents.

Evaluation of MSP2

â&#x20AC;&#x153;Otherâ&#x20AC;?

Informal Educator

Student

Technology Specialist

Researcher

Curriculum Coordinator

College Faculty

Librarian/Media Specialist

Participation Rank

Classroom Teacher

Table 8. MSP2 Website Questionnaire: Respondent Title by Participation Rank

3 or 4

As shown in Table 9, teachers who participated at higher levels typically had more years of teaching experience. Table 9. MSP2 Website Questionnaire: Respondent Teaching Experience by Participation Rank PR

Teaching Experience

10.6 years

13.7 years

16.0 years

3 or 4

17.4 years

Forty-five percent of mathematics teachers were categorized as non-participating, while only 15% of science teachers were rated as non-participants. Forty-percent of those who taught both science and mathematics participated at high levels (Table 10).

Evaluation of MSP2

Table 10. MSP2 Website Questionnaire: Respondent Subject Area by Participation Rank

Mathematics

Mathematics and Technology

4 (45%)

1 (100%)

2 (15%)

1 (25%)

2 (22%)

2 (40%)

4 (31%)

1 (25%)

1 (11%)

1 (20%)

4 (31%)

2 (50%)

3 or 4

2 (22%)

2 (40%)

3 (23%)

Total

Science and Mathematics

Science

Other

Content analysis was performed on open-response items for all respondents to make comparisons between levels of participation and attitudes towards web tools. Respondents with a PR of 0 or 1 were more likely to state that they were new to the MSP2 Ning social network. One respondent with a PR of 1 stated, “I haven’t been a member long. Many things I have not explored.” A majority of all respondents (63%) indicated that resources were one feature of the website that they found most helpful and used most often. A participant with a PR of 2 stated, “The resources feature is where I continue to go to when searching for curriculum related to Math and Science.” Blogs were identified most often (26%) as the least helpful and least used feature. Features indicated as most and least helpful were common among all respondents. Half of the respondents with a PR of 0 indicated that they had little to no previous experience with online social networking whereas only 18% of respondents with a PR of 3 or 4 expressed this lack of experience. “Time” was indicated most often as the reason for not participating actively on the MSP2 Ning and for not participating in online social networking in general. Other reasons given for lack of participation included fear of “sounding unprofessional,” lack of technical expertise, and “grave concerns about personal privacy and Big Brother.”

MSP2 Educator Questionnaire The MSP2 Educator Questionnaire was developed by Project Team members from the Educational Development Center (EDC) and made available online between April 30 and June 15, 2009. The EDC distributed the questionnaire link via partner websites (NMSA website, MSP2 Ning) as well as on their own networks such as the ITEST Learning Resource Center.8 A total of 617 respondents completed the questionnaire. A summary of the findings is provided here with the EDC’s analysis available in Appendix I. A majority of respondents (70%) were middle school educators in science or mathematics. When asked about technology use, a majority of the respondents (75%) indicated use of a computer in the classroom, while 37% of the respondents had used a SMARTBoard (or similar) in their classroom. Websites used with students included Brainpop,9 Discovery,10 and NASA.11 Respondents indicated that they found websites through search engines (56%) or from colleague, parent, or youth recommendations (27%). Respondents (18%) indicated a lack of available computers as an obstacle to using technology in their classrooms.

ITEST Learning Resource Center: http://itestlrc.edc.org/ Brainpop: http://www.brainpop.com 10 Discovery: http://discovery.com 11 NASA: http://www.nasa.gov 9

Evaluation of MSP2

The search engine, Google,12 was the website respondents indicated was preferred by students for educational or academic help. Others included Brainpop, Ask,13 Funbrain,14 and Wikipedia.15 Respondents indicated that website features most engaging to students included games (29%) and youth-regulated pacing (25%). When asked how their use of technology in the classroom impacted their students, most respondents indicated that the use of technology engaged and motivated students (19%) and provided another way of learning (12%).

MSP2 Youth Questionnaire The MSP2 Youth Questionnaire was developed by the Project Team and was made available online between April 30 and June 15, 2009. The questionnaire was distributed to students by participating middle school educators who accessed the link through partner websites (i.e., NMSA website, MSP2 Ning) as well as the EDC’s own networks such as the ITEST Learning Resource Center. A total of 440 student respondents completed the questionnaire. A summary of the findings is provided with the EDC’s analysis available in Appendix J. A majority of respondents were male (49%) and identified themselves as White (52%). Respondents were relatively evenly distributed among Grades 6, 7, and 8. Most respondents indicated that they had computer access (68%) and an Internet connection (65%) in their homes. Students indicated playing online games most often as a reason for using computers for entertainment and basic computer applications, such as word processing and email, as their most frequent educational use. Of those respondents who indicated using social networking sites (43%), more than half (58%) indicated logging on at least once a day. For educational purposes, student respondents indicated using computers for science courses monthly (35%) or weekly (28%), whereas for mathematics courses hardly ever (39%) or never (24%). Students indicated using Google most often for science or math schoolwork. Respondents indicated ease of navigation most often as both a characteristic of their favorite website as well as a characteristic that defined youth-oriented websites. Respondents indicated being comfortable using technology for many tasks (e.g., to organize information, to solve everyday problems, to communicate with other people). Many respondents (27%) agreed with the statement, “For the most part, I believe the information found on the Internet.”

Webmetrics While different components of the portal16 were active for varying lengths of time prior to project funding, the MSP2 Ning social network was launched in January 2009. Active components prior to project funding included Exemplary Resources for Middle School Math and Science Blog (February 28, 2008) and Connecting News with National Science Education Standards Blog (September 5, 2007). Webmetrics included in this evaluation report were collected between May 2009 and April 2010. Webmetrics reports included the number of visits to the site, how visitors found the website (traffic sources), and the average number of pages viewed during each visit. These data are described for each component including the Ning and were reported by quarter for Year 2.

Google: http://www.google.com Ask: http://www.ask.com 14 Funbrain: http://www.funbrain.com 15 Wikipedia: http://www.wikipedia.com 16 http://msteacher2.org 13

Evaluation of MSP2

MSP2 Ning Social Networking Website Webmetrics have been collected for the MSP2 Ning since its launch on January 1, 2009. Data analyzed for the Year 2 evaluation were collected between May 2009 and April 2010. The number of new visitors between May 2009 and April 2010 increased by 210% (Table 11). Direct traffic17 to the website decreased from the first quarter to the fourth quarter (from 34% to 26%). Visitors from referral sites also decreased during this time (from 38% to 25%). However, visitors referred by search engines increased significantly (from 27% to 49%) during the same time period. Average page views decreased from 3.9 pages to 2.5 pages between the first and fourth quarters. Table 11. MSP2 Ning Visits, May 2009 - January 2010 First Quarter (May – Jul 2009)

Second Quarter (Aug – Oct 2009)

Third Quarter (Nov 2009 – Jan 2010)

Fourth Quarter (Feb – Apr 2010)

Total Visits

5646

14114

12602

14131

New Visits

3331

9597

9073

10316

34%

29%

22%

26%

38%

33%

24%

25%

27%

39%

54%

49%

3.9

3.0

2.4

2.5

% of Visits from Direct Traffic % of Visits from Referring Sites % of Visits from Search Engines Average Pages Viewed per Visit

New and Returning Visitors Comparison Webmetrics data were analyzed for differences in MSP2 Ning site usage between new and returning visitors to help describe a profile of participation. Monthly usage trends were analyzed for number of visits, average page views per visit, and average time spent on the website. Figures 1 through 3 show these comparisons and indicate trends.

Number of Visits As shown in Figure 1, the number of visits increased for both new and returning visitors until August 2009. Then the number of visits for returning visitors leveled off while the number of visits for new visitors continued to increase until October 2009. The trend for the two groups was similar for the time period with somewhat more fluctuation for new visitors.

Direct traffic refers to website visits that were a result of going straight to the URL as opposed to being linked to the website through another website or search engine.

Evaluation of MSP2

Figure 1. Comparison of number of visits of new and returning visitors to MSP2 Ning, May 2009 â&#x20AC;&#x201C; April 2010.

Average Page Views per Visit Returning visitors, on average, visited twice as many pages per visit than did new visitors between May 2009 and April 2010. However, there was a decrease of about 40% in page views for both groups during this time (Figure 2).

Evaluation of MSP2

Figure 2. Comparison of average page views per visit of new and returning visitors to MSP2 Ning, May 2009 â&#x20AC;&#x201C; April 2010.

Average Time Spent on MSP2 Ning The average time spent on MSP2 Ning (Figure 3) was about 60% higher for returning members than for new visitors from May 2009 to April 2010. However, returning visitors average time on the site decreased (by 4:14 minutes) more than the average time for new visitors decreased (by 1:03 minutes). Average time on MSP2 Ning increased for both groups following winter academic breaks.

Evaluation of MSP2

Figure 3. Comparison of average time spent by new and returning visitors on MSP2 Ning, May 2009 â&#x20AC;&#x201C; April 2010.

Connecting News with National Science Education Standards Blog The Connecting News to the National Science Education Standards Blog18 was created for the first NSDL Middle School Portal project as a communication forum for middle school educators to disseminate and discuss current events and other related information as teaching opportunities. Since September 1, 2008, the MSP2 project has hosted the Connecting News with National Science Education Standards blog and periodically posted new topics based on current science news and events. Topics have included current events such as climate change and alternative energy and their connection to Grades 5-8 National Science Education Standards, as well as, suggestions for middle school inquiry lessons. The first blog entry was on September 5, 2007. As shown in Table 12, the total number of visitors to the Connecting News with National Science Education Standards Blog during Year 2 increased 42% from the first quarter to the second quarter, decreased 10% from the second to the third quarter, and then increased (8%) from the third to fourth quarter. These trends appeared to follow summer and winter academic breaks. The majority of visits to the blog were from new visitors. Traffic sources (i.e., visits from direct traffic, from referring sites, and from search engines) remained relatively consistent across the four quarters. Average pages viewed during each visit also changed very little.

http://expertvoices.nsdl.org/connectingnews

Evaluation of MSP2

Table 12. Connecting News to the National Science Education Standards Blog Visits, May 2009 January 2010 First Quarter (May – Jul 2009)

Second Quarter (Aug – Oct 2009)

Third Quarter (Nov 2009 – Jan 2010)

Fourth Quarter (Feb – Apr 2010)

Total Visits

4294

6098

5497

5911

New Visits

3950

5671

5112

5379

20%

15%

16%

19%

73%

82%

80%

76%

1.7

1.6

1.5

1.7

% of Visits from Direct Traffic % of Visits from Referring Sites % of Visits from Search Engines Average Pages Viewed per Visit

Exemplary Resources for Middle School Math and Science Blog The Exemplary Resources for Middle School Math and Science Blog19 was launched as part of the NSDL Middle School Portal project. The Exemplary Resources for Middle School Math and Science Blog has been hosted by the MSP2 project since September 1, 2008 to help middle school mathematics and science teachers find online resources for use in the classroom. The first blog entry was February 28, 2008. Table 13 shows that the total number of visitors to the Exemplary Resources for Middle School Math and Science Blog increased 69% from the first quarter to the second quarter, decreased 24% from the second to the third quarter, and increased again (6%) from the third to the fourth quarter. These trends appeared to follow summer and winter academic breaks. The majority of visits were from new visitors. Traffic sources remained relatively consistent across the four quarters. Over 83% of the visits were from search engines. Average pages viewed during each visit were consistent (2.3 pages) across the four quarters.

http://expertvoices.nsdl.org/middle-school-math-science

Evaluation of MSP2

Table 13. Exemplary Resources for Middle School Math and Science Blog, May 2009 -January 2010 First Quarter (May – Jul 2009)

Second Quarter (Aug – Oct 2009)

Third Quarter (Nov 2009 – Jan 2010)

Fourth Quarter (Feb – Apr 2010)

Total Visits

12007

20278

15376

16300

New Visits

10806

18250

13838

14507

11%

12%

85%

88%

87%

83%

2.3

% of Visits from Direct Traffic % of Visits from Referring Sites % of Visits from Search Engines Average Pages Viewed per Visit

MSP2 Wiki Pages As part of the MSP2 project, content originally created for the NSDL Middle School Portal project, and available online in static form, was migrated to a wiki and linked to the MSP2 Ning. This allowed MSP2 Ning members to modify resources from the NSDL website and to create new content. The MSP2 wikis were launched February 1, 2009. Webmetrics collected from May 2009 to April 2010 showed a dramatic increase in the total number of visits from the first quarter to the fourth quarter (from 1,574 to 19,621, Table 14). The number of new visits also increased from the first quarter to the third quarter proportionally (from 1,101 to 17,463). While visits from referring sites and direct traffic decreased from the first quarter to the fourth quarter, visits from search engines significantly increased (from 7% to 78%). Average pages viewed during each visit decreased from the first quarter to the fourth quarter (from 4.0 to 1.8 pages.)

Evaluation of MSP2

Table 14. MSP2 Wiki Page Visits, May 2009 - January 2010 First Quarter (May – Jul 2009)

Second Quarter (Aug – Oct 2009)

Third Quarter (Nov 2009 – Jan 2010)

Fourth Quarter (Feb – Apr 2010)

Total Visits

1574

14155

16970

19621

New Visits

1101

12456

15103

17463

32%

10%

61%

24%

16%

67%

74%

78%

4.0

2.0

1.8

% of Visits from Direct Traffic % of Visits from Referring Sites % of Visits from Search Engines Average Pages Viewed per Visit

Teacher Leader Interviews MSP2 Teacher Leaders (TL) were interviewed in March 2010. An Evaluation Team member conducted four interviews by phone using the MSP2 Teacher Leader Interview Protocol developed specifically for this project. Interview transcriptions were analyzed using NVivo for topic driven content analysis.

Teacher Leader Demographics Four Teacher Leaders (two male and two female) have teaching experience in science, mathematics, or technology between 9 and 26 years. All had extensive experience with both personal and professional Web 2.0 tools, including social networks. One TL recalled early use of educational technology stating, “I was one of the pioneers of using Internet in my classroom back in the very early 1990s when nobody could have an account unless it was attached to a university or the government.” Two TLs had participated in this project for 6 months, while the remaining two had participated for a full year.

Teacher Leader Responsibilities Interview transcripts revealed that Teacher Leaders saw their main responsibilities as recruiting, increasing participation, generating discussion, and presenting webinars for the MSP2 Ning. One TL discussed recruitment of new members stating, “I also [spent]… a lot of the time at the National Middle School Conference giving out stuff about MSP2, talking about MSP2, wearing my button and that kind of thing, trying to get more interest.” All four TLs spoke extensively about the responsibility of encouraging members to become active participants. Ways in which this was done are discussed in greater detail in this section under Participation. TLs reported generating discussion among members by sharing their own experiences and ideas used in the classroom. A TL described his aim for discussion prompts stating, “…the things I focus on are things that are readily available to teachers specifically, like shareware and freeware and things of that nature.” When asked about responsibilities of the position, one TL mentioned that he was scheduled to present a webinar shortly after the interview.

Evaluation of MSP2

Comparisons of Social Networks Teacher Leaders were asked to compare the MSP2 Ning with other social networks in which they had participated. Two TLs indicated that MSP2 Ning members were less active on the site when compared to members of other social networks. One TL framed the need to increase member participation as the next phase saying, “…we’ve got the numbers now we need to try to get enough topics flowing to keep the interest that are getting people comfortable enough to chat more, they’re still kind of shy in a lot of respects.” Another TL indicated that the MSP2 Ning was less active than other sites he was involved with because MSP2 Ning membership was much smaller than the others. One TL compared the MSP2 Ning to other social networks in terms of available resources. She stated: …the difference I noticed to this Ning as opposed to the other Nings, the other Nings, all the resources are what they have, where our Ning, becomes a link to other resources, so it actually channels in more information than just what’s on the Ning itself, so the resources go beyond just what the people themselves bring because there’s the connection to the digital library….

Participation Teacher Leaders were asked to characterize MSP2 Ning member participation. All four TLs indicated that member participation had “…a long way to go before it’s what we all envisioned….” One TL characterized member participation as voyeuristic stating, “…I would suggest that probably most of them just kind of go around and more or less lurk and kind of check things out….” The TLs indicated that they saw increased member participation over the course of their MSP2 Ning experience. One TL observed, “When I first started there wasn’t as much discussion, but I’ve seen more and more discussion….” TLs mentioned targeted emails and webinars, as well as welcome messaging, as effective strategies they used to motivate members to be more active on the Ning. One TL explained the welcome message strategy stating, “…the first step is getting them to feel like ‘I can play here,’ you know, so [we] welcome them right off the bat saying ‘hi’….”

MSP2 Ning Features Interview transcripts revealed that Teacher Leaders felt that the features most helpful to members included those that connected to the resources, such as digital library, webinars, and discussion forums. One TL mentioned peer discussions in which teachers expressed their appreciation for the MSP2 Ning Resource Guides. Three TLs mentioned discussion forums as a helpful resource. One TL explained how these worked and why they were helpful, stating: …from my perspective, the message boards, I know that as a teacher myself that’s the part that’s most helpful to me…I said, in the beginning of the school year, here’s the free and shareware that I use in my classroom, and then I talked a little about each piece, I’d go, ‘what do you use’ and I was able to get several new tools for my classroom just because I started the ball rolling but then others contributed…I can think of three specific pieces of software that I used this year that was directly impacted by what I got from our community. Two TLs mentioned the webinars as helpful features. One TL explained why she thought webinars were helpful, stating, “…the free webinars…I think that they are just tremendous resources and anybody who knows anything about webinars know that they are tremendously expensive generally, and for us to be able to offer these for free, I think is just terrific.” The other TL recalled webinars he hosted that had between 12 and 50 people attending.

Evaluation of MSP2

Improvements All four TL agreed that increased participation would make the MSP2 Ning community better. One TL described the site as “young and new” stating that, “…when the numbers improve, which they have steadily grown, I think the community will become more and more strong.” One TL noted that adding more content and events may “draw them in.”

Face-to-Face vs. Web-based Professional Development TLs mentioned time, collegiality, anonymity, sharing, and technological barriers as factors to be considered when providing professional development through a web-based format. Time was seen as both a disadvantage and an advantage. One TL thought conveying information via web-based communication took more time than face-to-face interactions. He explained this view stating: You can’t automatically address every specific question a teacher has…you just have to kind of take your time and chip away at it, like there’s been some times where we’ve helped out folks where if I was there I could have helped them over the course of 10 minutes, but because we’ve had to keep going back and forth to be more clear, more specific about a situation that 10 minute conversation may turn into 40 minutes worth of postings back and forth…. Two TLs saw time as an advantage. One TL felt “…the time it takes to just drive to the college and find a parking space, I thought, how much time am I wasting just doing that when I can do so much of my college work online, and that becomes one of the considerations of the 21st century.” Another TL considered teachers’ schedules stating, “…if somebody’s got a problem at 3 in the morning and they send it out, if there’s somebody on it then, they might get an answer at 3 in the morning whereas they certainly wouldn’t get an answer until tomorrow otherwise….” One TL regarded the decrease of collegiality within the school setting as a disadvantage of webbased professional development stating, “…you’re very collegial with the people you are connecting with socially online, but you don’t see them as closely… I’m not going down the hall to talk to the other teachers as much as I would’ve in the past to get ideas and things like that….” One TL mentioned technology as a disadvantage of web-based professional development. She discussed the balance between catering to members with more experience with Web 2.0 tools and those new to technology, stating, “…if you offer more tools for the more advanced learner, it becomes overwhelming and it frustrates the ones who are less familiar, and so it becomes a difficult balance.” She later added, “I don’t know if there is a way to make the Ning less intimidating, you can reduce tools but then that limits for other people.” TLs mentioned anonymity and sharing as advantages of web-based professional development. One TL stated, “…you don’t have that fear of failure…the anonymity tries to allev[iate] some of that.” Another TL discussed the sharing of ideas from different perspectives, “…I can start to see the innovative ideas that other states are doing, it’s like, why don’t we do that out here in [omitted], I can start to bring out ideas from other places that are a little bit different from what we’ve all heard and done over and over again, so, I like the fact that by having something that’s online you can connect to people from farther distances.” She later added that a member from Italy had attended a webinar event she hosted.

Member Problems or Concerns Teacher Leaders were asked about problems or concerns they may have addressed for MSP2 Ning members. The only concern expressed to one TL was regarding a lost password. The TLs

Evaluation of MSP2

could not remember any other instances in which they needed to address a problem or concern regarding use of MSP2 Ning.

Document Review Two middle-school educators reviewed MSP2 web-based resources using the Web-based Resource Review Rubric created by the Evaluation Team specifically for this project. All new MSP2 Resource Guides and at least 10% of new posts for each blog (Exemplary Resources for Middle School Math and Science and Connecting News with National Science Education Standards) were reviewed.20 Blog posts were randomly selected for review. Blog posts and Resource Guides were separated and reviewed independently for mathematics and science.

Resource Guides MSP2 Resource Guides were developed for this MSP2 project in September 2008, hence all MSP2 Resource Guides dated September 2008 or later were reviewed for this evaluation. Reviewers assessed the Resource Guides within their discipline, using the Web-based Resource Review Rubric, for connection to standards, relevance, adaptability, accuracy, currency, coverage, navigability, and appeal (Table 15). A middle-school science educator reviewed eight science Resource Guides and gave an average score of 3.66 (A-) to the resources. Navigability received the lowest score. Several links were noted as missing or broken. A middle-school mathematics educator reviewed the five mathematics Resource Guides and awarded an average score of 3.77 (A-). Research/Readings received the lowest score although the reviewer indicated that the category did not apply to some of the publications. Full reviews for each RESOURCE GUIDES can be found in Appendix K. Table 15. Resource Review Average Scores for Science and Mathematics MSP2 Resource Guides, Spring 2010 Criteria Standards Relevance Adaptability Accuracy Research/Readings Currency Coverage

Science A(3.83) A(3.67) A(3.83) A(3.83) A(3.67) A(3.83) A(3.67)

Mathematics A(3.83) A(3.83) A(3.67) A (4.0) B(2.60) A (4.0) A (4.0)

Resource items dated September 2008 or later are considered new for this project.

Evaluation of MSP2

Criteria Navigability Appeal Mean

Science B(2.83) A(3.75) A(3.66)

Mathematics A (4.0) A (4.0) A(3.77)

Blogs Two blogs were linked to MSP2 Ning. The Exemplary Resources for Middle School Math and Science blog has been hosted by the MSP2 Project since September 1, 2008. Fifty-two (21 science, 27 mathematics, and 5 technology) blog entries were posted between September 1, 2008 and January 13, 2010. The Connecting News to the National Science Education Standards blog also has been hosted by the MSP2 Project since September 1, 2008 although the first entry was not posted until October 8, 2008. Thirty-nine science themed blogs were created between this date and December 16, 2009. Reviewers randomly chose 10% of the blog entries within their discipline and using the Webbased Resource Review Rubric, assessed them for connection to standards, relevance, adaptability, accuracy, currency, coverage, navigability, and appeal (Table 16). Full reviews for each of the science and mathematics blogs can be found in Appendix L. Six science blog entries scored an average of 3.71 (A-). Reviewer comments revealed missed opportunities such as, “downgraded only due to the recent earthquake news omitted due to the lack of updating of site.” The reviewer also suggested discussing controversies tied to some of the themes in the blog entries such as “Origin of Species” and “Polar Bears and Climate Change” adding further comments such as, “a lack of response to the blogger who commented about the cold winter in Ohio indicating that global warming could therefore not be happening.” Seven mathematics blog entries were reviewed and scored an average of 3.31 (B+). Two categories received low scores: Research/readings and coverage. Comments indicated most of the blog entries reviewed did not mention research addressing the topic. The reviewer offered suggestions such as, “may want to show research showing the benefits of visual thinking in mathematics.” One suggestion regarding coverage stated, “as the blog posts expand in topic and frequency, it would be a good idea to connect to previous/future posts….” Table 16. Resource Review Average Scores for Science and Mathematics Blog Entries, Spring 2010 Criteria Standards Relevance Adaptability Accuracy Research/Readings

Evaluation of MSP2

Science A(3.67) A(3.67) A(3.67) A(3.67) A(3.50)

Mathematics B (3.07) A(3.67) A(3.57) A(3.57) C+ (2.14)

Criteria Currency Coverage Navigability Appeal Mean

Science A(3.67) A(3.67) A(3.83) A (4.0) A(3.71)

Mathematics A(3.83) C+ (2.21) A(3.71) A (4.0) B+ (3.31)

Thematic Website Review Beginning in August 2009, a monthly thematic website review was conducted on the MSP2 Ning to collect data describing quantity and quality of participation. Therefore, quantitative and qualitative data were collected and analyzed. In March 2010, the MSP2 Ning had 735 members, almost twice as many as in August 2009 (370). Members represented 23 countries, including the United States (681), Canada (10), and the United Kingdom (11), but also the Philippines (4), Bhutan (2), and Sudan (1). Between August 2009 and March 2010, four new Groups21 were added, making the total 11, with the membership of each group ranging from 4 to 63 members. The new groups were Working with English Speakers of Other Languages, Middle Childhood Science Methods, MSP2 Book Club, and Educational and Action Research. One of the original discussion groups, Integrating Technology, had the largest membership (63). Within this group, there were nine discussion topics, five of which were threaded.22 Topics with the most responses included What Kind of Freeware/Shareware do you use? (15 responses) and iPods (10 responses). Seven group discussions were randomly chosen and evaluated for quality of conversation using the MSP2 Social Network Conversation Rubric. Posts from Project Team members initiated three of the seven discussions. Project Team members, including Teacher Leaders, were very active participants in these discussions. Ning member participation, and therefore conversation quality, was low for all except one discussion in the Math Group called Math is the bomb! This discussion, regarding student “’hatred’ and fear” of math and how they are “reinforced by parents who don’t expect their kids to be good at math because they were never good at math,” was posted by a Ning member and involved 10 responses from seven participants. The participants shared opinions and suggestions for reducing the “stigma of math” creating a high- quality conversation. The number of forums did not change much between August 2009 and March 2010. One new forum was added, NMSA Conference Sessions. However, as shown in Table 17, the number of topics and responses within each forum increased during this time period.

MSP2 Groups are composed of MSP2 Ning members, who have formed a group around a common experience (e.g., Science, Math Teacher Education, At the District Level). 22 For the purposes of this evaluation, a threaded discussion is one in which there is more than one response to the main post.

Evaluation of MSP2

Table 17. Discussion Forum Topics and Replies, August 2009 and March 2010 Category Integrating Technology Teaching Science Teaching Mathematics Uncategorized NMSA Conference Sessions

Topics August 2009 6 1 2 5

Topics March 2010* 18 5 9 12

Responses August 2009 37 1 23 11

Responses March 2010* 75 27 55 40

*Cumulative totals.

Eleven forum discussions were randomly chosen and evaluated for quality of conversation using the MSP2 Social Network Conversation Rubric. Project Team members, including Teacher Leaders, posted 5 of the 11 discussion topics, 2 of which were notes and observations from conference meetings. Three of the 11 discussion topics (Social networking sites in the middle school classroom, Science fair ideas, and Need help on a transition unit for 5th graders moving to middle school!) were posted by participants and elicited high-quality, active conversations. MSP2 Ning blogs were removed in November 2009 due to illicit use but were added back to the website in February 2010 after stricter security processes were put in place. Number of blog posts, number of comments, and quality of comments were collected from August 2009 to November 2009 (Table 18). Twenty additional blog posts were added to the MSP2 blog in this 3month time period. Although the number of blog posts with responses increased, the percentage of those with more than two responses decreased from 80% to 46%. This decrease indicated that although blog posts were being responded to, the posts were not eliciting online conversation. The most popular blog posts, based upon number of responses, were Problem based learning meets needs? (22 responses) and So, I’m starting a STEM course… (13 comments). Table 18. MSP2 Blog Activity, August 2009 and November 2009. Activity Number of Posts Number of Posts with Responses Number of Posts with More Than Two Responses

August 2009 61 20

November 2009* 81 39

*Cumulative totals.

Between August and November 2009, six blog posts23 were evaluated using the MSP2 Social Network Conversation Rubric. Three of the posts were brief descriptions of webinars that had been held. These posts did not elicit quality conversation. Two of the posts So I’m starting a STEM course… and Technology for discussion, elicited medium-quality conversations involving Project Team members along with MSP2 Ning members. The blog post, Problem based learning meets needs? elicited high-quality conversation including robust sharing of classroom teacher participant experiences with the topic. Project Team members, including Teacher Leaders, were very active in all online conversations (group discussions, discussion forums, and blogs). Not only did they post the majority of the topics, their responses were quick, detailed, and many were geared toward eliciting further conversation. Member participation was low except for specific individual topics that seemed to 23

Only blogs that included more than two responses were evaluated.

Evaluation of MSP2

elicit high-quality conversation. Many resources are shared in discussions where participants and Project Team members share books and online applications that they have used in their classrooms. As shown in Table 19, Diigo24 members, bookmarks, and visits increased for each category (i.e., Math, Science, and Professional Resources) between August 2009 and February 2010.25 Math Diigo groups went from least to most popular during this time period. Table 19. Diigo Group Members, Bookmarks and Visits, August 2009 and February 2010 Group Math Science Professional Resources

Members August 2009 9 11

Members February 2010* 19 17

Bookmarks August 2009 93 39

Bookmarks February 2010* 106 43

Visits August 2009 91 59

Visits February 2010* 155 95

108

109

*Cumulative totals.

Dissemination Project Team members disseminated MSP2 project information at the following conferences, meetings, and online events:

Presentations at Local and National Conferences and Meetings • • • • • • • • •

ITSCO/TechColumbus Education Innovation Mixer (Columbus, OH) (5/6/09) School Library Journal Annual Meeting (Washington, DC) (10/1/09) National Middle School Association (NMSA) Annual Conference (Indianapolis, IN) (11/4-11/7/09) National Science Digital Library (NSDL) Annual Meeting (Washington, DC) (11/17-11/19/09) Climate Literacy Digital Network Conference Call (1/26/10) National Science Education Leadership Association Annual (NSELA) Meeting (Philadelphia, PA) (3/17/10) Society for Informational Technology & Teacher Education (SITE) International Conference (San Diego, CA) (3/29-4/2/10) National Girls Collaborative Project (Hartford, CT) (4/5/10) Collegiate Middle Level Association Annual Meeting (Westerville, OH) (4/17/10)

Promotional Materials Distributed at Conferences and Meetings • • •

Center of Science and Industry (COSI) Teacher Fair (Columbus, OH) (8/4/09) National Middle School Association (NMSA) Annual Conference (Indianapolis, IN) (11/4-11/7/09) Science Education Council of Ohio (SECO) Conference (Columbus, OH) (2/25-2/26/10)

Diigo is a social bookmarking website through which members can share favorite website links for similar topics. 25 Diigo group information was removed from the MSP2 Ning in February 2010.

Evaluation of MSP2

Webinars Project Team members presented a total of 14 webinars during Year 2 using the Elluminate Live! interface. Webinars were advertised through the MSP2 Ning, as well as the National Middle School Association (NMSA) website (www.nmsa.org), the Middle E-Connections electronic newsletter, and the What’s New from the NMSA electronic newsletter. At the conclusion of the webinars, participants were asked to complete a 7-item questionnaire evaluating their 26 experience. Data from eight webinars were available and a summary is included in this report. Webinar evaluation reports provided by the Project Team are included in Appendix M.

Webinar 1: Interactive Notebooks The first webinar of Year 2, Interactive Notebooks, was presented by Teacher Leader Todd Williamson on July 29, 2009. Although 19 participants attended the webinar, only four responded to the questionnaire. Of the four respondents, two indicated their roles as library, resource, or technology coordinator, one indicated teacher/classroom educator, and one indicated other. Two respondents indicated that they primarily worked in Grades 5-8. All four respondents indicated that the examples and illustrations were a beneficial aspect of the session; three respondents indicated that the presentation of the session was beneficial; and two respondents indicated the Q & A section was helpful. All four respondents rated the quality of the webinar as excellent. Respondents indicated that the examples of ways to use specific digital tools was the most useful information attained from the webinar. One respondent indicated having technical difficulties during the session. Other topics requested by respondents included Interactive Whiteboards, Twitter in the classroom, and Voicethreads.

Webinar 2: Moodle for Middle School On August 5, 2009 a second Year 2 webinar session, Moodle for Middle School, was presented by Teacher Leader Tom Jenkins and involved 52 participants; 11 respondents completed the evaluation questionnaire. Most of the respondents (73%) indicated their primary role as teacher/classroom educator, while 27% of the respondents indicated library, resource, or technology coordinator. Examples and illustrations was chosen most often (73%) as the most beneficial aspect of the session, followed by a tie between presentation and Q & A (27%). A majority of respondents (55%) rated the session as excellent, while 36% indicated a very good rating. One respondent rated the session as good. One respondent relayed “confidence to try Moodle” while another felt “encouraged” to use Moodle with middle school students as outcomes of the webinar. One respondent had experience with Moodle but was happy to learn “…some other ways of doing things.” Other respondents mentioned appreciation for links, resources, new information, and examples for trying Moodle in their classrooms. Most respondents (91%) did not have any technical difficulties during the session, but one respondent indicated otherwise. Respondents indicated that they would like to see more information on Web 2.0 tools in the classroom in future webinars.

Webinar 3: Digital Storytelling Digital Storytelling was the topic of the third Year 2 webinar. The webinar was presented by Project Team member Jessica Fries-Gaither on August 12, 2009. Fourteen participants joined the session of which half completed the evaluation. Most respondents (57%) were teachers/classroom educators. One respondent indicated researcher and two respondents indicated other as their role in education. Two respondents indicated they primarily worked in Grades 5-8, while the rest of the respondents (72%) indicated other. Examples and illustrations 26

No data were collected for Webinars 4, 6, 10, 11 and 12. Only one participant responded to the evaluation for Webinar 13.

Evaluation of MSP2

was chosen frequently (71%) as the most beneficial aspect of the session, followed by presentation (57%), Q & A (43%), and other (29%). A majority of the respondents (71%) rated the webinar as excellent, while one respondent indicated very good, and another indicated good. Links for teacher support, start up tips, practical ideas, and steps in developing digital stories were noted as helpful information from the session. None of the respondents indicated having technical problems during the webinar. Opportunities to share struggles and successes with digital storytelling, a session on Voicethread, and digital storytelling tools for visually handicapped students were indicated as ideas for future webinars.

Webinar 5: Interactive Whiteboards Fifteen participants attended the Interactive Whiteboards webinar presented by Teacher Leader Todd Williamson on September 16, 2009. Four participants completed the webinar evaluation. Half of the respondents indicated their primary role as library, resource, or technology coordinator, while the other half indicated either teacher/classroom educator or other. Each respondent indicated a different work environment (elementary grades (preK-4), Grades 5-8, school district, and other). All respondents indicated examples and illustrations as the most beneficial aspect of the session. Other beneficial aspects noted were presentation (67%) and Q & A (33%). Two respondents rated the session as very good and one respondent rated it excellent. One respondent indicated having technical difficulties during the webinar. One participant was “glad to see there are others out there who think the same as we do” and was glad to have “great resources.” Another participant was appreciative of “sites to visit for more information.” Using Google Apps in the classroom was suggested as a topic for a future webinar.

Webinar 7: Diigo: Social Bookmarking External Advisory Board member, David Hayward, presented the seventh Year 2 webinar on October 7, 2009. Of 18 participants, 6 completed the webinar evaluation questionnaire. A majority of respondents (67%) indicated their primary role as library, resource, or technology coordinator. The remaining respondents indicated other. Most respondents (67%) indicated other as the primary work setting, while one indicated Grades 5-8 and another indicated Grades 9-12. All respondents indicated presentation as the most valuable aspect of the webinar, while many indicated examples and illustrations (67%) or Q & A (50%). Respondents rated the webinar excellent (83%) or very good (17%). Respondents indicated that helpful aspects of the session included differences between Diigo and Delicious27, how to show students how to use Diigo for research, and how to use Diigo tools. None of the respondents indicated having technical difficulties. A follow-up session on how to use Diigo in the classroom was suggested as a topic for a future webinar event.

Webinar 8: UDL PDQ External Advisory Board member, Mindy Johnson, presented the eighth webinar on Universal Design for Learning (UDL) on October 21, 2009. Eleven participants attended and six completed the evaluation questionnaire. Half of the respondents indicated their primary role as administrator. One respondent indicated teacher/classroom educator and two indicated other. A majority of the respondents (83%) indicated their primary work setting as other, while one respondent indicated Grades 5-8, and another respondent indicated Grades 9-12. All respondents indicated presentation as the most valuable aspect of the webinar, while many also indicated examples and illustrations (67%). Half of the respondents gave the webinar a quality rating of excellent and the other half indicated very good. Respondents indicated that learning about free UDL tools available through the Center for Applied Special Technology (CAST),28 detailed information on UDL, and suggestions for getting additional help on the topic were the most 27 28

Diigo and Delicious are two examples of social networking tools. CAST: http://www.cast.org/about/index.html

Evaluation of MSP2

helpful aspects of the webinar. Half of the participants experienced technical difficulties during the webinar. Algebra software was the only suggestion listed for future webinars.

Webinar 9: Laboratory Safety National Science Teacher Association (NSTA) Science Safety Consultant, Ken Roy, presented the ninth webinar titled Laboratory Safety on November 12, 2009. Twelve participants attended and six completed the evaluation questionnaire. Half of the respondents indicated their primary role as teacher/classroom educator and one-third indicated student. A majority of the respondents (67%) indicated their primary work setting as Grades 5-8 while one indicated elementary grades (preK-4). Half of the respondents gave the webinar a quality rating of excellent and the other half indicated very good. Respondents indicated that misconceptions about laboratory safety were clarified and new safety procedures were learned from the webinar. One respondent “appreciated the pictures” that could be used to identify hazards. One-third of the participants experienced technical difficulties during the webinar. Respondents suggested possible topics for future webinars such as ways of integrating Project Wild and Project Wet into state science standards and examples of lessons and hands-on activities to use in the classroom.

Webinar 14: Digital Tools and Math Twenty participants attended the Digital Tools and Math webinar presented by Teacher Leaders Tom Jenkins and Eric Beiderbeck on March 23, 2010. All participants completed the webinar evaluation. Most of the respondents (40%) indicated their primary role as teacher/classroom educator while 25% indicated library, resource, or technology coordinator. Half of the respondents indicated other as their primary work environment with the remaining respondents distributed relatively evenly amongst Grades 5-8, Grades 9-12, and school district. Most of the respondents (65%) indicated examples and illustrations as the most beneficial aspect of the session. Other beneficial aspects noted were presentation (50%) and Q & A (5%). Over half of the respondents (58%) gave the webinar a quality rating of excellent, 37% indicated very good, while one respondent rated the webinar quality as good. One respondent indicated having technical difficulties during the webinar. Fourteen participants indicated that they learned about multiple new digital applications and tools that can be used in the mathematics classroom. One participant indicated this by stating, “Very interesting applications I was not aware of, I would like to try to incorporate it in my learning environments. I would also like to try and introduce it to my children’s teachers.” Suggestions for future webinar topics included connecting classrooms across borders and more specific math tool usage instructions.

Tapped - In Events MSP2 Project PI, Kimberly Lightle, hosted two online chats utilizing Tapped In29 meeting space. On June 8, 2009, six participants joined a chat that focused on the wiki-based resource guides from the MSP2 Ning and on August 10, 2009, four participants joined a chat about the MSP2 Ning social network.

Tapped In: http://www.tappedin.org

Evaluation of MSP2

External Advisory Board Meeting The second MSP2 External Advisory Board (EAB) meeting was held on July 15, 2009. The virtual meeting was hosted through the Elluminate Live! interface. Eleven EAB members and eight Project Team members attended the online event. Project Team members presented progress reports for each of the project goals. Project PI, Kimberly Lightle, moderated the session and began by reviewing the five project goals. She then updated participants on navigational and content modifications to the site (Goal 1), followed by an update on participation and activity in the MSP2 Ning social community (Goal 2). Project Team member Mary Henton described the Teacher Leader activities for promoting MSP2 Ning participation. Sarita Pillai of the EDC spoke on progress made with development of the Virtual Learning Environment (VLE). She described youth and educator focus group data, Design Team members and activities, and displayed examples of work and video of the Design Team work sessions. The Goal 3 report included a description of conferences, partnerships, publications and online events and that was followed by a brief discussion of sustainability (Goal 4). A few evaluation activities (Goal 5) and findings were addressed to conclude the presentation. External Advisory Board members offered their thoughts and suggestions. One member advised creating a “best practices” document and provided suggestions for how to encourage Ning members to become “organizers of conversations.” Another EAB member advised endorsing the site at the district level. He suggested identifying a district or two to introduce the site so that school-based leaders could act as promoters of the website. Another EAB member suggested that organizing resources by topic might be a way to make district-level participation more accessible. A Project Team member noted that survey data could be used to find topics that could serve as district-wide topics. Finally, the topic of sustainability was addressed by an EAB member. She suggested identifying aspects of the project on which to focus, “what aspects do you want to sustain and why?”

Evaluation of MSP2

Continuing Evaluation Activities Ohio State University project personnel will continue to develop, modify, and catalogue quality learning resources for the MSP2 project. Usage statistics for the portal components will be collected on a quarterly basis and posted to the project wiki for continuous, formative evaluation of the project. Resources will be reviewed by external experts utilizing tools and protocols developed in Year 2. The National Middle School Association (NMSA) will continue to host monthly webinars that promote the MSP2 project. Reports, including registration and survey data, will be posted to the wiki for team member and evaluator review. NMSA project personnel also will continue to recruit and train Teacher Leader candidates. A randomly selected subsample of Teacher Leaders will be interviewed annually. The Education Development Center (EDC) will finalize youth Virtual Learning Experiences (VLEs) and post the completed product to the MSP2 Ning. The products will be reviewed by external experts and formative feedback will be provided to the Project Team in Year 3. All Project Team members will finalize sponsorship and e-commerce avenues so as to sustain the MSP2 Ning after the final year of funding. Semi-annual External Advisory Board meetings will continue to support this goal. The next step in the evaluation process for the E & A Center is to investigate impact of the MSP2 Ning on members. Follow-up Teacher Leader interviews will be conducted to gain an internal perspective on the changes in Ning participation and activity. Members of various participation ranks will be interviewed to collect data on impact of varying levels of use of the MSP2 Ning. VLEs will be reviewed for appeal and access to middle-school students. All data will be synthesized to provide periodic and summative feedback to the Project Team.

Evaluation of MSP2

Summary and Recommendations Upon review of several sources of project data, findings of the Year 2 evaluation for the Middle School Portal 2: Math & Science Pathways (MSP2) Project are as follows: •

Learning resources have been developed, selected and modified, organized and contextualized into the MSP2 Ning. The resources are available to visitors and members via the MSP2 Ning through components of the Ning such as blogs, wikis, and resource links. The development of new content for the Ning has exceeded the goal for Year 2 and the NCS Search Service has been integrated so that the resource collection can be searched. Resource lists, blog posts, and wiki pages appear to be effective means for contextualizing learning resources for middle-school educators. Experts in the fields of science and mathematics middle-school education found the resources to be relevant, accurate, and appealing. (Goal 1)

•

The MSP2 Ning continues to provide access to content and interactive opportunities through blogs, wikis, discussion forums, and resource links. Resources can now be searched through the MSP2 Ning. Virtual support for educators is being provided through the Ning by a growing number of Teacher Leaders. Quality and quantity of virtual interactions will be evaluated further during Year 3. While the VLE templates have been finalized and implementation is scheduled to occur soon, the Ning does not offer activities for youths at this time. (Goal 2)

•

Resource discovery and usage have been promoted through MSP2 dissemination activities, including 13 webinar presentations that impacted 306 participants. Launch activities for new resources and online events through digital tools such as e-newsletters and Twitter have been utilized. Search engine data, monitored through webmetrics, showed that the MSP2 Ning and components continue to be located by users. Profile data indicated that the target audience has been reached. (Goal 3)

•

A second Advisory Board Meeting was held to discuss project progress on tasks and goals. Collaboration and cooperation has continued although numerous monthly conference calls have been canceled and the project wiki has become so large that information can be difficult to find and organize. The Project Team has pursued ideas related to marketing, branding and sustainability. Project partners continue to work diligently on their assigned tasks and to communicate progress or obstacles. (Goal 4)

•

Focus group data and questionnaire data from educators and youth, and youth Design Team data have been used to develop the VLE template. A profile of participation has been identified from data collected during Year 2. Questionnaires and interview protocols are being developed to collect impact data from teachers and students during Year 3. Impact data also will be collected by monitoring the VLEs to assess the quantity and quality of virtual interactions among users. (Goal 5)

•

Extensive users of the MSP2 Ning tend to be female science teachers with more than 10 years of teaching experience, as well as, previous experience with social networking. Teachers tend to use the Ning to find resources for use in the classroom. Teachers use the Ning more often during the academic school year, although they tend to spend more time and view more pages when visiting the website during the summer academic break. (Goal 5)

Evaluation of MSP2

Based upon evaluation data and findings to date, the Evaluation Team makes the following observations: â&#x20AC;˘

The MSP2 Project Team should continue to purposefully and thoughtfully develop and/or modify and migrate products and resource materials to the site. Most resources have been judged by participating teachers and external experts to be high-quality, relevant, and important resources for teaching mathematics and science. Time and effort should be invested in developing and making accessible those resources that are most needed and used by the target audience. Project components that cannot be maintained without considerable allocation of resources (e.g., blog posts) should be carefully assessed for the value they add to the site.

â&#x20AC;˘

While resources have been well-received and frequently accessed via the MSP2 Ning, the interactive components of the site are less so. The Project Team should continue to collect and explore data regarding the Ning user profiles and use these data to determine marketing strategies that are most appropriate to meet project goals. The Project Team should determine which of the following is a priority: (a) attracting and retaining the most active users of the site, (b) attracting and engaging users who are currently not active on the site, or (c) a combination of both approaches. Ideally, the project would engage and retain the active participation of many types of users but as project resources diminish it may not be possible to do so. Discussions about sustainability should focus on the goal of determining the priority audience and applying targeted strategies to reach and increase the participation of these individuals and groups in order to promote a site that meets project expectations for interactivity and user knowledgesharing.

Evaluation of MSP2

References Ohio Resource Center for Mathematics, Science, and Reading. (n.d.). ORC resource review rubric. Retrieved March 15, 2010 from: http://www.ohiorc.org/about/rubric.aspx

Evaluation of MSP2

Appendices Appendix A: Profile Questions Survey .................................................................................... 39 Appendix B: MSP2 Website Questionnaire ............................................................................. 40 Appendix C: MSP2 Educator Questionnaire ............................................................................ 45 Appendix D: MSP2 Youth Questionnaire ................................................................................ 48 Appendix E: MSP2 Teacher Leader Interview Protocol ............................................................ 58 Appendix F: Web-Based Resource Review Rubric ................................................................... 60 Appendix G: MSP2 Participation Rank Rubric ......................................................................... 61 Appendix H: MSP2 Social Network Conversation Rubric ......................................................... 62 Appendix I: MSP2 Educator Questionnaire Analysis ................................................................ 63 Appendix J: MSP2 Youth Questionnaire Analysis .................................................................... 74 Appendix K: MSP2 Web Resource Score Sheet: Resource Guides ............................................ 91 Appendix L: MSP2 Web Resource Score Sheet: Blogs ............................................................. 98 Appendix M: Webinar Analyses ............................................................................................105

Evaluation of MSP2

Appendix A

Evaluation of MSP2

MSP2 Website Questionnaire

Page 1 of 1 Appendix B

Evaluation of MSP2 MSP2 Website Questionnaire Fall, 2009 Dear Participant: We want to thank you for your participation in the MSP2 Website Questionnaire. This online questionnaire is designed to obtain information about your knowledge and use of the MSP2 Social Network website. Although you will not receive immediate benefit from completing this questionnaire, the information obtained will be useful for improving this website. The questionnaire takes no more than 15 minutes to complete. Although we have asked for identification information in order to link your responses across the points of data collection, you will never be identified in any reports or summaries of the data. After individual responses are entered into the database, access to your responses is strictly limited. Your participation is completely voluntary; you may refuse to answer certain questions or withdraw from the evaluation at any point. All the questionnaire data will be confidential. Failure to participate will not affect you in any way, but it will weaken the overall study because your important ideas and opinions will not be represented. By clicking to the next page, you indicate your consent to participate in this portion of the evaluation. If you have questions about the questionnaire or evaluation, please contact me at 513-529-1686. If you have questions about participant rights, please contact the Office for the Advancement of Research and Scholarship at Miami University, 513-529-3600. Thank you again for your participation. Sincerely, Sarah B. Woodruff, Director Ohioâ&#x20AC;&#x2122;s Evaluation & Assessment Center for Mathematics and Science Education

Next >>

Miami University, Oxford, OH 45056. www.muohio.edu. 513.529.1686

Evaluation of MSP2

https://survey.muohio.edu/Checkbox/Survey.aspx?s=8897533e97cf4490a31df6850a4a9f5b

5/26/2010

MSP2 Website Questionnaire

Page 1 of 1

Evaluation of MSP2 MSP2 Website Questionnaire Fall, 2009 Instructions: This questionnaire contains five demographic items and six open-ended items. Please provide responses that best describe your opinions and activities. Please DO NOT click the "Finalize the Questionnaire" button until your responses are really finalized. Clicking "Finalize the Questionnaire" will submit your report and you will no longer be able to access it. If you have any questions about using this online survey system, please contact Yue Li (liy@muohio.edu).

*1. The first letter of your FIRST name is:

*2. The first letter of your LAST name is:

*3. Your date of birth is: (Format: MM/DD/YYYY)

4. Are you a member of the MSP2 project team or advisory board? Yes

5. Are you an MSP2 teacher leader? Yes

<< Back

Next >>

Miami University, Oxford, OH 45056. www.muohio.edu. 513.529.1686

Evaluation of MSP2 https://survey.muohio.edu/Checkbox/Survey.aspx?s=8897533e97cf4490a31df6850a4a9f5b

41 5/26/2010

MSP2 Website Questionnaire

Page 1 of 2

Evaluation of MSP2 MSP2 Website Questionnaire Fall, 2009 1. How did you find the MSP2 website?

2. What features of the MSP2 website attracted you and prompted your decision to joi MSP2 social network?

3. What features of the MSP2 website were helpful to you and did you use most freque after joining (e.g., resources, blog, wiki)?

Evaluation of MSP2

https://survey.muohio.edu/Checkbox/Survey.aspx?s=8897533e97cf4490a31df6850a4a9f5b

5/26/2010

MSP2 Website Questionnaire

Page 2 of 2

4. What features of the MSP2 website were not helpful to you and did you not use afte joining (e.g., resources, blog, wiki)?

5. What has been your previous experience with personal or professional social networking? What are your reasons for choosing to engage in or not engage in social networking?

6. What suggestions do you have to improve the MSP2 website?

<< Back

Next >>

Miami University, Oxford, OH 45056. www.muohio.edu. 513.529.1686

Evaluation of MSP2

https://survey.muohio.edu/Checkbox/Survey.aspx?s=8897533e97cf4490a31df6850a4a9f5b

5/26/2010

MSP2 Website Questionnaire

Page 1 of 1

Evaluation of MSP2 MSP2 Website Questionnaire Fall, 2009 7. If you are willing to be contacted by Ohio's Evaluation and Assessment Center, please provide your name and email address below:

<< Back

Finalize the Questionnaire

Miami University, Oxford, OH 45056. www.muohio.edu. 513.529.1686

Evaluation of MSP2

https://survey.muohio.edu/Checkbox/Survey.aspx?s=8897533e97cf4490a31df6850a4a9f5b

5/26/2010

Appendix C

(MSP)2 Educator Questionnaire Questions [page 1] Required

Welcome to the (MSP)2 Survey! If you’ve reached this page, then that means you have been asked you to fill out a survey. By filling out this survey, you will help us learn about middle school youth and how they use the Internet. The questions on this survey are about: what youth think about science and math, whether or not you use computers and the Internet in your classrooms or programs, how youth use the Internet, and what youth look at when they go online. We are trying to design a Website for middle school youth, and we hope you’ll help us. If you take the time to fill out this survey, your answers will help us design our Website based on your feedback, and we will better understand what middle schoolers like and dislike on the Internet. It won’t cost you anything to take the survey—it’s free! Also, we will have a drawing in June 2009 and 10 participants will get a $50 Amazon.com gift certificate as a way of thanking you for the time you give us if you participate. If you fill out your contact information at the end of the survey, we’ll enter you in the drawing and send you your $50 Amazon.com gift certificate if you are picked. We will do everything we can to keep the information you tell us in the survey private. The information you give us will only be seen by project staff at Education Development Center, Inc., and we will never use your name or anything that could give away who you are. We will keep all our records confidential. Additionally, this survey is completely voluntary: you are in no way obligated to complete the survey once you have started. Questions about the (MSP)2 survey should be directed to Sarita Pillai, at 1-800-225-4276 ext. 2164 (spillai@edc.org). If you have any questions regarding your rights as a participant in this survey, you can contact EDC’s Human Protection Administrator at 1-800-225-4276 ext. 2971 or HumanProtections@edc.org. STATEMENT OF CONSENT I have read the summary of this survey above. By clicking on the “Take the Survey!” button below, I am saying that I would like to participate in the (MSP)2 survey. [Click here to start the (MSP)2 survey] [page 2] Both questions required. What subjects do you currently teach or does your program currently address? Check all that apply. Science: _______________ Math: _________________ Other: ________________

Evaluation of MSP2

(MSP)2 Educator Questionnaire Questions What grade level(s) do you currently work with? Check all that apply. 5th grade 6th grade 7th grade 8th grade [page 3] All questions optional. What technologies (e.g., computer, digital camera/camcorder, smart board) do you use with your youth? Open-ended, paragraph What Websites do you use with your youth? Open-ended, paragraph How do you usually find these sites? Open-ended paragraph Do your students have any Website preferences for educational/academic help? Check one. Yes No Please list these Websites here: Open-ended, list up to 5 Indicate how each of the following describes the ways your youth learn. a. Youth group projects often involve technology usage. Check one. Strongly agree Agree Disagree Strongly disagree b. Youth develop their own â&#x20AC;&#x153;technology learning environmentâ&#x20AC;? by engaging their educator, fellow youth, and experts outside of the classroom. Check one. Strongly agree Agree Disagree Strongly disagree [page 4] All questions optional. In your experience thus far, what have you found to be most valuable in terms of using the computer/Internet with your youth? For instance, have you found that certain kinds of Websites

Evaluation of MSP2

(MSP)2 Educator Questionnaire Questions engage your youth more than other kinds of Websites? Or perhaps you have found particular elements of certain Websites (games, colors and font, etc.) that your youth especially gravitate towards? Open-ended, paragraph In your experience thus far, what have you found to be the most frustrating, least useful about using technologies with your youth? Open-ended, paragraph As you look forward, what do you anticipate as the most likely obstacle(s) when using computers/Internet with your youth? Open-ended, paragraph Describe the impact of your work using computers/Internet with youth on your youths’ achievement in math or science. Open-ended, paragraph Please list some of the topics you use to teach science or math concepts (e.g., stock market, glaciers and global warming). Open-ended, list up to 5 [page 5] All questions optional.

Thank you for taking the (MSP)2 survey! If you would like to enter yourself in the drawing for a $50 Amazon.com gift certificate, please enter your contact information in the lines below. This information will only be used if you are one of the 10 participants to win a $50 Amazon.com gift certificate. We will use this information to let you know that you’ve won and send your $50 Amazon.com gift certificate to you. Phone: Open-ended, one line Email: Open-ended, one line Questions about the (MSP)2 survey should be directed to Sarita Pillai, at 1-800-225-4276 ext. 2164 (spillai@edc.org). If you have any questions regarding your rights as a participant in this survey, you can contact EDC’s Human Protection Administrator at 1-800-225-4276 ext. 2971 or HumanProtections@edc.org.

Evaluation of MSP2

Appendix D

Youth Questionnaire Questions [page 1] Required

Welcome to the (MSP)2 Survey! If you’ve reached this page, then that means your educator or mentor has asked you to fill out a survey. Why should I fill out this survey? By filling out this survey, you will help us learn about middle schoolers (such as yourself) and how they use the Internet. The questions on this survey are about: what you think about science and math, whether or not you use computers and the Internet, how you use the Internet, and what you look at when you go online. We are trying to design a Website for middle school youth like you, and we hope you’ll help us. If you take the time to fill out this survey, your answers will help us design our Website based on your feedback, and we will better understand what middle schoolers like and dislike on the Internet. How much does it cost? Do I get anything? It won’t cost you anything to take the survey— it’s free! Also, we will have a drawing in June 2009 and 10 participants will get a $25 Amazon.com gift certificate as a way of thanking you for the time you give us if you participate. If you fill out your contact information at the end of the survey, we’ll enter you in the drawing and send you your $25 Amazon.com gift certificate if you are picked! We’ll be sure to keep your contact information separate from your survey, so that we are sure to use that information only to contact you if you’ve won the raffle. It’s completely up to you! We also want you to know that if you agree to fill out the survey and then you change your mind later, that’s okay. Nothing bad will happen to you if you decide not to participate or you change your mind later. It’s your choice. CONFIDENTIALITY: We will do everything we can to keep the information you tell us in the survey private. The information you give us will only be seen by us (the people who made the survey), and we will never use your name or anything that could give away who you are. We will do a good job at keeping all our records secret by following the rules that the U.S. government has made for researchers. Do you have any questions? Questions about the (MSP)2 survey should be directed to Sarita Pillai, at 1-800-225-4276 ext. 2164 (spillai@edc.org). If you have any questions about your rights as a participant in this survey, you can contact EDC’s Human Protection Administrator at 1-800225-4276 ext. 2971 or HumanProtections@edc.org. STATEMENT OF ASSENT I have read the summary of this survey above. By clicking on the “Take the Survey!” button below, I am saying that I would like to participate in the (MSP)2 survey. [Click here to start the (MSP)2 survey]

Evaluation of MSP2

Youth Questionnaire Questions

[page 2] All questions optional. First, we want to know how you feel about using computers and the Internet to complete tasks. Pick the answer that best represents your agreement or disagreement with each statement. I am good at collecting information using technology (e.g., Internet, search engines). Check one. Strongly agree Agree Neutral Disagree Strongly disagree I am good at using technology to organize information (spread sheets, word processing). Check one. Strongly agree Agree Neutral Disagree Strongly disagree I know how to use technology to solve everyday problems. Check one. Strongly agree Agree Neutral Disagree Strongly disagree I avoid using technology because I am not very good at using it. Check one. Strongly agree Agree Neutral Disagree Strongly disagree I am good at using technology to better understand ideas that I learned in school. Check one. Strongly agree Agree Neutral Disagree Strongly disagree I am good at using technology to communicate with other people. Check one. Strongly agree Agree

Evaluation of MSP2

Youth Questionnaire Questions Neutral Disagree Strongly disagree If I could use the technologies that scientists and engineers use when they do their work, it would be easier for me to learn. Check one. Strongly agree Agree Neutral Disagree Strongly disagree [page 3] All questions optional. Next, we have some questions about the kinds of jobs youâ&#x20AC;&#x2122;re interested in. List one job you think youâ&#x20AC;&#x2122;d like to have as an adult. Open-ended; one line Do you think you will need to use science, math, or technology to do this job? Check one. Yes No Not sure List up to five different jobs you aware of that involve science, math, technology, or engineering. Open-ended; list up to 5 What are some of your favorite science topics to learn about? Open-ended; list up to 3 What are some of your favorite math topics to learn about? Open-ended; list up to 3 Outside of school, how often did you visit museums, science centers, colleges, or do other activities (such as community service/volunteering or special programs) that present science, math, or technology? Check one. Never 1-2 times 3-5 times More than 5 times per year List some of these places below. Open-ended; list up to 5

Evaluation of MSP2

Youth Questionnaire Questions

List five of your favorite things to do for fun. Open-ended; list up to 5 [page 4] All questions optional. The next set of questions are about how you use the computer and what you do online. Where do you most often use a computer? Check one. Home School Library Other: _____________ Does your school have computers available for students to use? Check one. Yes No How often do you use computers in the following school courses? a. Science Check one. Never Hardly ever About monthly Weekly About daily b. Math Check one. Never Hardly ever About monthly Weekly About daily Do you think more of your classes could be taught using the Internet? Check one. Yes No Not sure How comfortable are you using computers? Check one. Not at all comfortable A little comfortable Comfortable Very comfortable Do you have a computer at home? Check one.

Evaluation of MSP2

Youth Questionnaire Questions Yes No If above is yes: Do you have Internet access at home? Check one. Yes No [page 5] All answers optional. We’re interested in knowing how you use computers and the Internet when you’re doing something for school and when you’re doing something for fun. If you only do an activity for school, just check the “For School” box. If you only do an activity for fun, just check the “For Fun” box. If you do an activity for school and for fun, check both boxes. Check all that apply. For For Fun Don’t Do School Create or work on my own page on social networking Websites (ex. MySpace, Facebook) Create my own characters/avatars and do things in virtual worlds (ex. Second Life, WOW, Gaia, or Habbo Hotel) Send or receive instant messages Talk to people in “chat rooms” Download or share music/video files Create or work on webpages Write computer programs Use software packages to create animations Use a digital camera of digital camcorder to capture images or video and upload them online Share something online that you created yourself, such as your own artwork, photos, music, stories or videos Take material you find online or that you’ve created—like songs, text or images—and remix it into your own artistic creation Using search engines to find useful information on the Internet Use large databases to find or analyze information Use computers to analyze data Use basic computer applications to write, research, and communicate (ex., Word, spreadsheets, email, presentations, Internet) Create or work on an online journal or blog Contribute to online forums, online dialogs, wikis Download a podcast so you can listen to it or view it at a later time Develop video games

Evaluation of MSP2

Youth Questionnaire Questions Play online games Use simulations Use integrated learning labs (“Tech labs”) What are the different ways you use social networking Websites (for example, Facebook or MySpace)? Do you ever use those Websites to… Check all that apply. Make new friends Stay in touch with friends you see a lot Stay in touch with friends you rarely see in person Flirt with someone Make plans with your friends Don’t use Other: ____________ If anything but “Don’t use” is checked: We’d like to know the specific ways you communicate with your friends using social networking Websites (for example, Facebook or MySpace). Do you ever… Check all that apply. Post messages to a friend's page or wall Send a bulletin or group message to all of your friends Send private messages to a friend within the social networking system Wink, poke, give "e-props" or kudos to your friends Post comments to a friend's blog Other: __________________ About how often do you visit social networking Websites (for example, Facebook or MySpace)? Check one. Several times a day About once a day 3-5 days a week 1-2 days a week Every few weeks Less often Not sure Do you ever use any of the following to play games, whether or not you personally have one? Check all that apply. A game console like an Xbox, a Playstation or a Wii A portable gaming device like P-S-P, D-S or Gameboy A desktop or laptop computer A cell phone or handheld organizer Don’t use If anything but “Don’t use” is checked: Do you play Massive Multiplayer Online Games, such as World of Warcraft? Check one. Yes No

Evaluation of MSP2

Youth Questionnaire Questions Not sure Do you play console games that utilize online multiplayer functions, such as Xbox Live? Check one. Yes No Not sure What are your current top three favorite games? Open-ended; list up to 3 [page 6] All answers optional. What Websites do you like to visit for fun? (URL or name of site) Open-ended; list up to 3 What Websites do you visit for school? (URL or name of site) Open-ended; list up to 3 What Websites do you visit when you want to find something on your own (not school related)? (URL or name of site) Open-ended; list up to 3 From all these Websites you visit which ones do you enjoy the most? (URL or name of site) Open-ended; list up to 3 [page 7] Are your favorite Websites…? Check all that apply. Easy to navigate Easy to find info Visuallly appealing Pleasant to interact with/interactive Youth oriented Other: ___________________ What does the description “youth oriented” mean to you? Check all that apply. Website that allows interactive content Website that is flashy Website that has a lot of images Website that is very colorful Website that is easy to navigate Other: _____________________________

Evaluation of MSP2

Youth Questionnaire Questions Weâ&#x20AC;&#x2122;re interested in the kinds of things you do when you use the Internet. Not everyone has done these things. Please just tell me whether you ever do each one, or not. Do you ever... Check all that apply. Buy things online, such as books, clothing or music Look online for health, dieting, or physical fitness information Go online to get news or information about current events Go online to get information about a college, university or other school you are thinking about attending Go to Websites about movies, TV shows, music groups, or sports stars you are interested in Watch a video on a video-sharing Website like YouTube or GoogleVideo Read the online journals or blogs of others How do you determine if a Website is trustworthy and the information provided is correct? Open-ended; paragraph For the most part, I believe the information found on the Internet. Check one. Strongly agree Agree Neutral Disagree Strongly disagree Do you visit Websites for science and math work? Check one. Yes No If the above is yes: a. Which Websites? Open-ended; list up to 3 b. Do you like these Websites? Check one. Yes No If above is answered: Why or why not? c. How did you find out about these Websites? Open-ended; paragraph [page 8] All answers optional. The last set of questions are questions about you. Current Grade Check one. 6 7 8 9

Evaluation of MSP2

Youth Questionnaire Questions What year were you born? Open-ended; 1 line What is your gender? Open-ended; 1 line What is your race? Check all that apply. Asian/Pacific Islander Black/African-American Hispanic/Latino American Indian White/Caucasian Other: _____________ What language do you speak at home? Open-ended; 1 line Is there anything else you would like to add about yourself or your interests? Open-ended; paragraph [page 8]

Thank you for taking the (MSP)2 survey! If you have questions about the (MSP)2 survey you just took, contact Sarita Pillai, at 1-800-2254276 ext. 2164 (spillai@edc.org). If you have any questions regarding your rights as a participant in this survey, you can contact EDCâ&#x20AC;&#x2122;s Human Protection Administrator at 1-800-225-4276 ext. 2971 or HumanProtections@edc.org. If you would like to enter yourself in the drawing for a $25 Amazon.com gift certificate, please click on the link below and enter your phone number and email. This information will only be used if you are one of the 10 participants to win a $25 Amazon.com gift certificate. We will use this information to let you know that youâ&#x20AC;&#x2122;ve won and send your $25 Amazon.com gift certificate to you. [survey monkey link]

[new survey] All answers optional.

Thank you for taking the (MSP)2 survey!

Evaluation of MSP2

Youth Questionnaire Questions If you would like to enter yourself in the drawing for a $25 Amazon.com gift certificate, please enter your contact information in the lines below. This information will only be used if you are one of the 10 participants to win a $25 Amazon.com gift certificate. We will use this information to let you know that youâ&#x20AC;&#x2122;ve won and send your $25 Amazon.com gift certificate to you. Phone: Open-ended; 1 line Email: Open-ended; 1 line

Evaluation of MSP2

Appendix E F

MSP2 Teacher Leader Interview Protocol 2009-10 Evaluation

“Thank you for taking the time to talk with me today. The data that I collect from these interviews will be used in the evaluation of the MSP2 project. The interview should take no longer than 30 minutes to conduct. At any time you may skip a question or if needed, you can ask to terminate the interview. This interview will be recorded so that I can focus on our conversation while having an accurate record of what we discussed. Do I have your permission to use your answers as data for the MSP2 project evaluation? Do I have your permission to tape our interview?” How long, and in what capacity, have you been in education (grade level, subject, etc)? Approximately how many social networks are you, or have you been, a member of? How long have you been an MSP2 Teacher Leader? What prompted your decision to become an MSP2 Teacher Leader? What responsibilities do you have as an MSP2 Teacher Leader? Based upon your experience with digital learning tools and social networking, how does the MSP2 Ning compare with similar sites? How would you characterize participation by members of the MSP2 Ning? What features of the MSP2 Ning appear to be most helpful to teachers? In what ways can the MSP2 Ning be improved? In your opinion what is the most valuable aspect of the Ning and why? What strategies have you used to motivate members to be active participants?

Evaluation of MSP2

What have you learned as an MSP2 Teacher Leader? What are the advantages and disadvantages of providing support to teachers via the NING versus in face-‐to-‐face settings? What problems or concerns have teachers expressed regarding the MSP2 Ning?

Evaluation of MSP2

Appendix F

Web-‐based Resource Review Rubric

Alignment with standards

Relevance

Adaptability

Accuracy

Research/Readings

A The resource has a direct and explicit link to content standards (state or national) (Describe in comments) The resource is clearly relevant to the target audience. (e.g., middle school math teachers, graduate students, etc) The resource is adaptable to a variety of settings and there is discussion that helps teachers adapt the resource to different settings. The content of the resource is accurate, no obvious factual or other errors. The resource clearly includes references to research or additional readings that discuss benefits of the content.

The resource is not well aligned with content standards.

The resource is not of the nature that standards would apply. (Explain in comments)

The resource does not apply to the target audience.

Does not apply.

The resource is adaptable to a variety of learning settings but suggestions for adaptation are largely absent.

The resource has a narrow range of use.

Does not apply.

The content is not inaccurate, but may not include desirable details or examples.

The content contains factual or other errors

Does not apply.

The resource reflects research that is widely known and generally accepted.

The resource is antithetical to commonly accepted research findings. (Explain in comments)

The resource is not of the nature that research would apply. (Explain in comments)

The content is slightly outdated.

The content is severely outdated.

The content is of the nature that currency does not apply. (Explain in comments)

The resource does not reference standards but would clearly contribute to student learning of one or more content standard. Some of the resource is relevant to the target audience while other parts may expand beyond.

Currency

The content is timely and up-to-date.

Coverage

The number of topics, depth of information and links to other information is valuable and not overwhelming.

The content would benefit from additional information or deeper discussion.

Navigability

The resource is easy to navigate and all links are active and up-to-date.

The resource is navigable without serious issue.

Appeal

The resource is appealing to the target audience. (e.g. saves time for teachers, appeals to middle school students)

The resource is somewhat appealing to the target audience.

The resource does not go into detail or add depth to the discussion OR the amount of topics and additional information could be considered overwhelming. The resource presents some identifiable navigation issues (e.g., inactive links, circular navigation, etc.) The resource is not appealing to the target audience.

Does not apply.

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Evaluation of MSP2

Participant has done ONE of the activities:

Little Participation 1

Participant has done nothing but joined the website.

No Participation

OR started 2-‐3 conversations by posting an initial and participating in the blog or group discussions

OR started a conversation by posting an initial and participating in a blog or group discussion

OR posted 3-‐4 thoughtful* responses

OR posted 1-‐2 thoughtful* responses

OR started 3 or more conversations by posting an initial and participating in the blog or group discussions

OR hosted an event.

OR posted 5 or more thoughtful* responses

Participates Extensive User Frequently 3 4 Participant does Participant does 7 4-‐6 of the listed or more of the activities (may be listed activities all in one or in (may be all in one multiple or in multiple categories): categories):

Participates Occasionally 2 Participant does 2-‐3 of the listed activities (may be all in one or in multiple categories):

*For the purposes of this evaluation, a “thoughtful response” is defined as a contribution that expands on the conversation and elicits further dialogue.

Discussion Initiator

Personalization Personalized “My Page” Comments Responded to original welcoming message. Group Membership Joined a “group.” Blogs/Discussions Added a comment to a blog or group discussion. Events Attended or “plans to” attend an event

Activities

MSP2 Participation Rank Rubric

Appendix G

Appendix H

Evaluation of MSP2 Ning Social Network Conversation Rubric

High

Medium

Low

Quality of Responses

Responses are relevant, in-depth and wellthought-out. They expand on the conversation and elicit further discussion.

Responses are relevant but short. Some do not further the discussion.

Responses not always relevant. They are consistently short and do not further the conversation.

Number of Responses

Participants contribute many responses and develop deeply threaded dialogues.

Participants contribute several responses in several conversations.

Participants contribute few responses.

Participant involvement

Responses are well distributed among participants, and all are deeply engaged in the discussion.

Responses are distributed between participants and Project Team members (including Teacher Leaders)

Conversations dominated by Project Team members (including Teacher Leaders) and only a few participants.

Source: iDiscovery Learning Community Scoring Rubric

Evaluation of MSP2

Appendix I

MSP2 Educator Questionnaire Analysis Between Thursday, April 30, 2009 and Monday, June 15, 2009, an online questionnaire for middle school educators was active. This survey, created and administered by Education Development Center, Inc. with help from the National Middle School Association and various partner networks was made available to middle school science and math educators across the nation. A total number of 631 educators attempted to respond to the questionnaire; of these 631 potential participants, 617 decided to go forward with and participate in the research study. The following is an analysis of the data collected via this questionnaire. Educator Demographics Participants primarily taught science or math, though a substantial percentage of participants taught both science and math. In addition, a handful of educators taught subjects other than science or math or were administrators. See Table 1. Table 1. Educator Demographics by Subject Taught Subject Science Math Science and Math Technology Science and Technology Other* No Answer TOTAL

N 238 247 93 4 2 17 16 617

% 39% 40% 15% <1% <1% 3% 3% 100%

*including Administrators, Gifted & Talented, Special Education, ESL, Art, Health, Social Studies, ELA, Library Science Participants also taught across middle school grades, with most teaching 6th-8th grades. See Table 2. Table 2. Educator Demographics by Grade Taught Grade 5th Grade 6th Grade 7th Grade 8th Grade Missing

N 45 246 307 331 15

%* 7% 40% 50% 54% 2%

*sums to greater than 100% due to teaching multiple grades

Evaluation of MSP2

Finally, five participants were Special Education educators, and three participants were Gifted and Talented educators. Technologies Used in the Classroom Participants were first asked about the technologies they used in their classrooms. Seventy-five percent of the participants used computers in their classrooms in some way. Large numbers of participants also employed SMARTBoards (or similar, such as Promethean Board, Mimio, InterWrite or ActivBoard), digital cameras/camcorders, and projectors/overheads. It seems that a small group of educators has also begun to incorporate the use of calculators, ELMOs, polling tools, tablets, and digital data collection tools as well. It should be noted that multiple variations in combinations of these technologies were given. See Table 3. Table 3. Technologies Used in the Classroom Technology Computer SMARTBoard (or similar) Digital Camera/Camcorder Projector/Overhead Calculator (4-fn, scientific, or graphing) ELMO/SMARTCam Polling Tool Tablet Data Collection Tool(s) DVD Digital Microscope VCR Scanner GPS TV iPod Webcam Digital Sound Recording Instrument CD Player LaserDisc Phone Missing

N 463 230 179 146 84 74 44 39 26 18 16 10 9 8 6 5 4 3 2 1 1 119

%* 75% 37% 29% 24% 14% 12% 7% 6% 4% 3% 3% 2% 1% 1% 1% 1% 1% <1% <1% <1% <1% 19%

*sums to greater than 100% due to use of multiple technologies

Evaluation of MSP2

Websites Used with Youth Participants were next asked about the Websites they used with the youth they work with. An abundance of specific sites were named, from science/math exclusive sites to general learning sites to Web tools to educator’s own sites. In addition to specific Websites, other, general types of Websites were also mentioned (see Appendix A). The top five specific Websites mentioned were: • BrainPop (73 mentions) • Discovery.com (49 mentions) • NASA (49 mentions) • United Streaming (31 mentions) • FunBrain (28 mentions) It should be noted that a large number of textbook sites were utilized as well as sites/domains provided to educators by states, counties, districts, or schools. Additionally, the prevalent use of search sites, YouTube, and Wikipedia by educators is something to also note. In addition, participants were asked about the methods by which they have come to identify the sites they use with their youth. Over half of the participants mentioned taking the time to search for Websites themselves. Yet another large group of participants relied on sites recommendations from colleagues, parents, or even the youth they teach. See Table 4. Table 4. Methods of Finding Websites for Use With Youth Method Search Recommendation by Colleagues, Parents, or Youth Professional Development Provided by State/County/District/School Textbook Resource Recommendation by Organization or other Website Missing

N 348 165 73 40 30 9 145

%* 56% 27% 12% 6% 5% 1% 24%

*sums to greater than 100% due to use of multiple methods Participants were then asked their thoughts on the most frustrating and least useful aspects of using technology in the classroom. The first and foremost frustration mentioned was the lack of computers/technology available to educators for whole class instructions (one computer for the whole class, one computer per student, multiple computers for students to share). Second to the availability of computers and technology was the fact that when technology was available, it was outdated, broken, or unable to connect to the Internet. See Table 5.

Evaluation of MSP2

Table 5. Frustrating Aspects of Using Technology in the Classroom

Lack of Computers Available Outdated/Broken Technology Lack of Reliable Internet Access Time to Start Up, Set Up Non-User-Friendly Time to ID Quality Sites Potential Distractions Laziness Spurred in Youth General Access Issues for Youth (lack of access at home) Blocking by School/District Variation in Youth Tech Knowledge/Skill Broken Links Lack of Financial Support Lack of Youth Media Literacy Lack of Support by Administrators Lack of Educator Tech Skills Cheating/Difficult to Determine Cheating Lack of Appropriate Online Behavior Does Not Substitute for Actual Labs, Movement Unable to Keep up with Evolving Technology Does Not Add to Quality of Education Timed Logins Don't Accommodate Slow Readers Missing

N 113 87 78 47 47 40 32 30 30 30 27 23 12 11 10 9 8 4 2 1 1 1 186

%* 18% 14% 13% 8% 8% 6% 5% 5% 5% 5% 4% 4% 2% 2% 2% 1% 1% 1% <1% <1% <1% <1% 30%

Finally, participants were asked to describe potential future obstacles they might encounter when using technology in their classrooms. Compared to the answers given regarding current frustrating aspects of technology in the classroom (Table 5), participants had dissimilar thoughts about future obstacles. The one consistent worry was the lack of computers/technology available to educators for whole class instructions (one computer for the whole class, one computer per student, multiple computers for students to share). Educators see general access issues for youth, their own lack of technology skills, lack of appropriate online behavior, lack of financial support, and inability to keep up with evolving technology as more future concerns than current concerns. In addition, they see their current frustration with outdated technology, unreliable Internet access, and variation in youth technology knowledge and skill as less frustrating in the future. See Table 6. Table 6. Likely Future Obstacles When Using Technology in the Classroom

Evaluation of MSP2

Lack of Computers Available General Access Issues for Youth (lack of access at home) Time to Start Up, Set Up Potential Distractions Outdated/Broken Technology Lack of Reliable Internet Access Time to ID Quality Sites Lack of Educator Tech Skills Lack of Appropriate Online Behavior Lack of Financial Support Blocking by School/District Unable to Keep up with Evolving Technology Laziness Spurred in Youth Variation in Youth Tech Knowledge/Skill Lack of Youth Media Literacy Lack of Staff Support to Address Issues Timed Logins Don't Accommodate Slow Readers Does Not Substitute for Actual Labs, Movement Cheating/Difficult to Determine Cheating Will Have to Incorporate Lessons on Online Etiquette Off-Hour Work Missing

N 118 73 50 49 47 41 36 31 30 29 23 15 14 13 13 5 3 3 2 2 1 189

%* 19% 12% 8% 8% 8% 7% 6% 5% 5% 5% 4% 2% 2% 2% 2% 1% <1% <1% <1% <1% <1% 31%

*sums to greater than 100% due to mention of multiple obstacles Youth Website Preferences Participants were asked if they noticed any particular Websites preferences among their youth for specifically educational or academic help. 158 participants did notice particular preferences in their youth while 341 participants did not noticed any particular preferences. Nonetheless, when asked about Websites they find their youth drawn to for educational or academic help, educators were able to name a substantial amount of Websites. Again, Websites mentioned ranged from specific science and math sites to general learning sites to educator’s own sites (see Appendix B). The top five specific Websites mentioned were: • Google (26 mentions) • BrainPop (19 mentions) • Ask.com (16 mentions) • FunBrain (16 mentions) • Wikipedia (15 mentions)

Evaluation of MSP2

Clearly, there are similarities and differences between the above sites and the ones mentioned in the “Websites Used with Youth” section above. While it seems that two of the top five specific sites intentionally used by educators as teaching tools with youth have seeped into youth usage, youth primarily rely on search sites (Google, Ask) or Wikipedia to identify educational or academic material online. Participants were also asked about particular Websites features they may have noticed that draw or engage youth more than other features. Two features in particular, inclusion of games and the ability for users to control their own pace as they go through the Website, stood out from the other features mentioned. See Table 7. Table 7. Features of Websites That Engage Youth Feature Games Allows User to Go at Own Pace Color Images Provides Correct, Relevant, Up-to-Date Content Video Provides Immediate Feedback Animation User-Friendly/Age Appropriate Less to Read Generally Engaging Web 2.0 Features Provides Easy Access to Information Music/Sound Easy to Read Font (large, black) Social Networking Search Layout Missing

N 176 155 51 46 35 35 33 32 25 19 17 14 13 12 10 9 3 2 194

%* 29% 25% 8% 7% 6% 6% 5% 5% 4% 3% 3% 2% 2% 2% 2% 1% <1% <1% 31%

*sums to greater than 100% due to mention of multiple features Ways in Which Youth Learn Participants were asked to indicate whether their youth learn in two particular ways: the integration of technology into group projects and the development of “technology learning environments” by youth.

Evaluation of MSP2

Over two-thirds of the participants indicated that technology was integrated into group projects; overwhelmingly, most educators do incorporate some type of technology into their curriculum. See Table 8. Table 8. Group Projects Involve Technology

Strongly Agree Agree Disagree Strongly Disagree Missing TOTAL

N 151 275 71 13 107 617

% 24% 45% 12% 2% 17% 100%

While high numbers of educators do incorporate technology, fewer educators identified the next step in technology integration: a more cohesive learning environment that engages youth, educators, and those outside of the classroom via technology. While still over half of the participants responded positively to this item, a greater percentage of participants responded negatively to this item. See Table 9. Table 9. Youth Develop "Technology Learning Environments"

Strongly Agree Agree Disagree Strongly Disagree Missing TOTAL

N 63 267 146 27 114 617

% 10% 43% 24% 4% 18% 100%

Impacts of Technology Use on Youth Participants were asked to describe the impact of their work using technology on their youthsâ&#x20AC;&#x2122; achievement in science or math. A majority of respondents stated that they have noticed their technology use increases youth engagement with the material they are trying to teach. Indirectly, this raises levels of youth achievement as engaged youth are more likely to excel. Educators who mentioned this shared both anecdotal evidence as well as scientific evidence from their own school districts, schools, or classrooms. The next largest impact mentioned was that technology provides another way for youth to learn (via multimedia, interactive, and social networking avenues)â&#x20AC;&#x201D;and so allows youth multiple ways to grasp material or demonstrate their understanding of material. Educators also found value in the immediacy of current and extensive information, the availability of online resources as added enrichment outside of the classroom, and the

Evaluation of MSP2

cultivation of 21st Century skills in youth. Finally, it should be noted that some educators still have found little to no to negative impact on their youth. Table 10. Impact of Technology Use in Classroom on Youth

Engages Youth With Material/Motivates Youth Provides Another Way of Learning General Positive Impact Provides Immediate Access to Current Info, More Info, and Feedback Enrichment for Classroom Lessons Cultivates 21st Century Skills Provides "Real Life" Applications Helps Educators Teach (organization, preparation) Allows Youth to Learn at Their Own Pace Demonstrates That School Can Be Cool Teaches That Technology is a Tool Provides Outlet for Creativity in STEM Accelerates Understanding Increases Participation in Class Connects Youth to Other Youth Little/No/Negative Impact Missing

N 118 77 50

%* 19% 12% 8%

44 38 29 17 15 15 9 8 7 4 4 4 27 204

7% 6% 5% 3% 2% 2% 1% 1% 1% 1% 1% 1% 4% 33%

*sums to greater than 100% due to mention of multiple impacts Science and Math Topics Finally, participants were asked to list some of the topics used to teach science or math concepts. While the complete list is broken into science and math (and further divided into specific and general topics) in Appendix C, the top five science and math topics mentioned are listed as follows: Science • Weather/meteorology (56 mentions) • Genetics/heredity (37 mentions) • Global warming (36 mentions) • Human body/anatomy/body systems (35 mentions) • Cells (33 mentions)

Evaluation of MSP2

Math • • • • • • •

Stock market (33 mentions) Fractions (26 mentions) Shopping (21 mentions) Coordinate planes/graphing (20 mentions) Percents (19 mentions) Ratios/proportions (19 mentions) General geometry (19 mentions)

Appendix A. Websites Used With Youth

Evaluation of MSP2

Appendix B. Youth Website Preferences

Evaluation of MSP2

Appendix C. Science and Math Topics

Appendix J

MSP2 Youth Questionnaire Analysis Between Thursday, April 30, 2009 and Monday, June 15, 2009, an online questionnaire for middle school youth was active. This survey, created and administered by Education Development Center, Inc. with help from the National Middle School Association and various partner networks was made available to middle school youth across the nation via their science and math educators. A total number of 445 youth attempted to respond to the questionnaire; of these 445 potential participants, 440 decided to go forward with and participate in the research study. The following is an analysis of the data collected via this questionnaire. Demographics A majority of youth who responded to demographic items were male (48%, N=213), White (35%, N=153), and spoke English at home. Most youth who responded to a question about birth year (31%, N=136) were born in 1995/1996; therefore the general age of participants was approximately 13-14 years. Fifty percent of participants responded to questions about their grade level; of those who responded, the majority were in 6th-9th grades (see Figures 1-4 and Table 1).

Table 1. Participants, by Home Language N %

Evaluation of MSP2

English French German Greek Hindi Japanese Korean Kurdish Oromo Punjabi Somali Spanish Vietnamese English/Bangla English/Cantonese English/German English/Japanese English/Korean English/Kurdish English/Nuer English/Spanish English/Yiddish English/Russian/Uzbek Missing TOTAL

176 1 2 1 1 1 1 1 2 1 1 3 1 1 1 1 2 1 1 1 10 1 1 228 440

40.00% 0.23% 0.45% 0.23% 0.23% 0.23% 0.23% 0.23% 0.45% 0.23% 0.23% 0.68% 0.23% 0.23% 0.23% 0.23% 0.45% 0.23% 0.23% 0.23% 2.27% 0.23% 0.23% 51.82% 100.00%

Participants were also asked about their favorite activities for leisure. Participants cited a range of activitiesâ&#x20AC;&#x201D;primarily sports and social activities, but games, jobs, art, academic work, and outdoor activities were readily cited as well. See Appendix A for a complete list of leisure activities. Computer Accessibility When asked about computer availability in their homes, 68% (N=300) participants stated that they had computers in their homes (as compared to 3%, N=13, participants who did not). In addition, when asked about Internet access in their homes, 65% (N=287) participants stated that they did have Internet access in their homes (as compared to 3%, N=14, participants who did not). Most participants who had computer access in their homes also had Internet access as well. When asked about computer availability in their schools, 75% (N=329) participants stated that there were computers available and 2% (N=9) participants stated that there were not (29% missing, N = 129).

Evaluation of MSP2

Participants were asked about their computer use. First, participants were asked what settings they primarily used the computer. Overwhelmingly, participants primarily used the computer in their own homes (see Figure 5).

Technology Use A section of the questionnaire was dedicated to learning exactly how youth use the Internet, both for school and for fun. Table 2 displays the results of this section.

Evaluation of MSP2

It was found that in general middle school youth are less likely creators and more likely consumers of particular content (such as video games, computer programs, animations, blogs). Unsurprisingly, youth reported using basic applications to write, research, and communicate and using the Internet for searching as their primary uses of the Internet for school. They also used the Internet to analyze data or information. For fun, however, it seems that youth use the Internet for other purposes, primarily for playing online games, chatting with friends, and downloading music or videos. This is in addition to using the Internet in the same ways they use it in school—meaning that youth are more Internet-savvy than we are allowing them (or they feel allowed) to be in school/for school or that youth do not see other functions of the Internet as aspects that can be related to school or schoolwork. Youth participants were most interested in looking at Web sites that had video capability and information about popular culture (including movies, TV shows, and music). See Table 3 for other ways in which youth used the Internet. Table 3. Internet Interest, by Site Type Watch a video on a video-sharing Website like YouTube or GoogleVideo Go to Websites about movies, TV shows, music groups, or sports stars you are interested in Go online to get news or information about current events Buy things online, such as books, clothing or music Go online to get information about a college, university or other school you are thinking about attending Read the online journals or blogs of others Look online for health, dieting, or physical fitness information Missing

175

39.77%

172

39.09%

149 119

33.86% 27.05%

66 58

15.00% 13.18%

51 218

11.59% 49.55%

*Total exceeds 100% as respondents were able to indicate multiple activities.

Youth participants were also given a series of questions regarding the Web sites they used most often for particular purposes (fun, school work, sources of information) and the sites they most enjoyed. Table 4 highlights the top three types of Web sites mentioned by participants for each of the questions posed (see Appendices B-E for specific Web sites mentioned). As demonstrated by Table 4, youth participants rely on “search” Web sites for most Internet needs. In addition, youth participants see social networking sites as not only fun or enjoyable sites, but as sources of information. Youth understand that game sites are not necessarily good sources of information for school or otherwise. Finally, these youth also know to look at sites provided by teachers/schools/school districts for schoolwork; in particular, specific sites for STEM

Evaluation of MSP2

subjects were identified the most. These sites, however, lacked association with being fun or enjoyable. It should be noted, though, that many sites mentioned in response to the “Web sites Used for School” question may have been mentioned because “ALL the good ones [“fun” Web sites] are blocked [in school].”

Social Networking Further probing regarding social networking was also included in the questionnaire. First, youth were asked about the frequency of social networking site use. With a 43% response rate, 25% (N=111) of participants stated they log into social networking sites at least once a day; approximately half of these participants log into these sites several times per day. Other respondents remained evenly spread over logging in over less frequent amounts of time (over spans of days or weeks; see Figure 6).

Evaluation of MSP2

Youth were asked about the particular different ways in which they used social networking sites, and 44% (N=195) of participants stated that they did use these sites (percentage of site users differs from above due to separate questionnaire items with differing response rates). Of the 44% that use social networking sites, the following information was obtained: • 71% post messages to friends’ pages • 67% send private messages (similar to email) to friends • 52% post comments on friends’ blogs or pictures • 38% send group messages (blanket messages) • 28% send indirect messages (“poke,” “wink,” “kudos”) to friends • 6% use chat functions to talk to friends* • <1% use sites for gaming with friends* • 3% did not provide data *These ways of using sites were written in by participants and were not offered as choices in the questionnaire.

These data demonstrate that youth readily use many of the communication features of social networking sites; they do not only use these sites to create their own pages and put themselves out in the world—they use these sites as a means to reach out to others as well. Online Gaming Youth were also asked questions about online gaming and their gaming preferences. We first asked participants what types of gaming devices they use, regardless of whether or not they own these devices themselves. Of the 64% (N=273) of participants who responded: • 54% of participants use a gaming console • 50% of participants use a computer • 42% of participants use a cell phone • 39% of participants use a handheld/portable gaming device. • 1% of participants did not use any of the above It appears that participants who participate in online gaming use a wide variety of devices to do so; there is not one particular device that sticks out as being used more than others.

Evaluation of MSP2

When asked whether they played multi-player games (78% response rate; N=343), 29% (N=129) of participants said they did play them, 43% (N=190) said they did not play them, and 5% (N=24) were unsure (percentage of site users differs from above due to separate questionnaire items with differing response rates). The top three participant games are: â&#x20AC;˘ Call of Duty â&#x20AC;˘ Halo â&#x20AC;˘ Guitar Hero Most participants were overwhelmingly drawn to action games, followed by sports games; only a few participants thought of academic games as their favorite (see Figure 7, see Appendix F for a detailed list of games).

Technology Use and STEM Computer use in courses, specifically in science and math courses, was also polled. In science courses, participants stated that they used the computer about monthly; however, in math courses, participants stated that they used the computer hardly ever (see Figure 8 and Figure 9).

In addition, a large number (45%, N=197) of participants thought that their classes could be taught using the Internet (see Figure 10).

Evaluation of MSP2

When asked whether they visited Web sites for science and math work, 36% of participants (N=157) said that they do, and 17% (N=74) said that they do not (48% missing, N = 209). Those that said they visited Web sites for science and math work said that they visited the following sites most often (see Appendix G for a full list of Web sites): • Google • Ask.com • Study Island Google by far had the most mentions. The high frequency of search engine use by participants indicates that no one site has been identified as “the” STEM site to go to. This is also confirmed by the high use of Study Island and other teacher/district sites: youth remember sites provided to them by teachers and schools; the participants in this study have not identified any specific STEM site as a widely known “go-to” site for STEM schoolwork. Youth participants proved to be adept at using the Internet and available Web tools— and they felt that use of these tools could occur more often in the classroom. One participant shared, “I think that everyone in schools everywhere should have a required school email address or AIM account. That way, you can ask teachers questions without raising your hands, or make less noise in the room. But everything could be monitored [sic], to make sure everyone is on task.” Of the participants who do use Web sites for science and math work, 28% (N=125) like the sites they use, and 4% (N=17) do not like the sites they use (68% missing, N=298). Some listed reasons for liking sites include: • Helps find information for school work/Useful • Fun • Helps participants learn Some listed reasons for disliking sites include: • Boring • Not used often • Relates to school/Participant dislikes school or STEM subjects

Evaluation of MSP2

A common answer to this question is reflected in this quote from one participant: “Because, science and math bore me mostly. Unless you're talking Star Trek. If they made more science things related to Star Trek and Spock and all that, I'd learn more and never take my attention off the lesson once.” More participants listed reasons why they do like STEM sites they use than participants who listed reasons why they disliked those same sites. It seems that participants understand that these sites provide them with knowledge, skills, and tools that are valuable to at least their immediate academic situation, if not future academic and/or career goals. For example, one participant wrote, “They try to [sic] hard to get kids to like learning. The career I plan to pursue in the future involves science and math, my least favorite subjects, but I know I'll get used to them.” A full list of reasons can be found in Appendix H. Technology Preferences Youth were asked multiple questions about their Web site preferences. One item identified particular characteristics of Web sites and asked youth to respond if they found those characteristics to be true of their favorite Web sites. From a 50% (N=222) response rate, it was found that: • 40% of favorite Web sites were easy to navigate • 31% of favorite Web sites had information that was easy to find • 30% of favorite Web sites were visually appealing • 27% of favorite Web sites were interactive • 21% of favorite Web sites were “youth oriented” • 3% of favorite Web sites were entertaining or fun* • 1% of favorite Web sites provided social networking/personalization options* • <1% of favorite Web sites were game oriented or provided games* • <1% of favorite Web sites were comical* • <1% of favorite Web sites looked good/were well made* • <1% of favorite Web sites had cool names* • <1% of favorite Web sites were “awesome”* • <1% of favorite Web sites were colorful* • <1% of favorite Web sites had music* *These favorite characteristics were written in by participants and were not offered as choices in the questionnaire.

Of the choices offered to participants, many favorite Web sites were easy to navigate; it seems as if this site feature is something that middle school youth are drawn to, though they do not actively seek it out as part of determining whether or not they enjoy a Web site. Of youth-given responses, most were that site content was entertaining or fun. This indicates that content is the primary draw for youth, as opposed to usability. When asked to define “youth oriented” by indicating either a given choice, set of given choices, or their own answer, participants regarded “youth oriented” sites as those that

Evaluation of MSP2

are easy to navigate (27%, N=118) and interactive (24%, N=107). Of participants supplying their own definition of â&#x20AC;&#x153;youth oriented,â&#x20AC;? the vast majority (7%, N=30) stated that content of the site was appropriate for youth or developed specifically for youth use (see Table 4 for greater detail). Table 5. Definitions of "Youth Oriented" N %* Easy to navigate 118 26.82% Interactive content 107 24.32% Colorful 90 20.45% Lots of images 89 20.23% "Flashy" 64 14.55% "For kids" content** 30 6.82% Where you can learn 1 0.23% something** Missing 217 49.32% *Total exceeds 100% as respondents were able to indicate multiple definitions. **Youth-given response.

Self-Efficacy with Technology Participants were asked about their comfortability with using computers. A majority of participants (66%, N=290) were either very comfortable or comfortable using computers (see Figure 11).

Participants were asked to assess their ability to use computers and the Internet to complete tasks. In general, youth feel confident in their ability to use technology in different capacities (collecting information, organizing information, communicating) and different aspects of their lives (school and out of school). More youth remained neutral about whether using technologies that scientists and engineers use would aid in their learning. See Figures 12-18.

Evaluation of MSP2

In sum, youth participants proved to be familiar with and adept at using the computer and the Internet in their everyday lives. One participant stated, “I love getting on computers to do things[.] I think it is very fun and great[;] if I didn’t have computers it would make my life a lot harder.” Web site Credibility Participants were asked to give a likert scale rating of their belief of information provided on the Internet. While 23% (N=101) of participants remained neutral on this item, the vast majority of those who did express an opinion one way or the other (27%, N=117) stated that they strongly agree or agree with the statement “For the most part, I believe the information found on the Internet.” See Figure 19 for greater detail.

The high number of “neutral” responses demonstrates a high uncertainty among middle school youth in their ability to determine credible or non-credible online content. In addition, participants were asked an open-ended question about how they determine whether or not sites are trustworthy. It was found that participants interpreted this question in one of two ways: to mean how they determined that the source and content of the site is credible or to mean how they determined whether the site was “Web safe” with regard to private or personal information. Most participants provided answers

Evaluation of MSP2

pertaining to the former, stating that they relied on the recommendation of the site by others (including peers, teachers, and parents) or they used triangulation of information to determine the accuracy of the content provided. Participants who answered the latter question primarily discussed visual cues displayed on the site itself demonstrating that it is a “safe” Web site. See Appendix I for detailed participant responses to this item. In sum, it was found that most participants either use tactics that are ineffective when trying to determine the accuracy of information on a Web site or multiple tactics to ensure that information is correct. For example, one participant stated, “I guess if it just basically sounds right. I don't use Wikipedia[;] that stuff can be wrong. I usually use Google, which is VERY trustworthy.” Identifying information that “sounds right” and relying on information about particular Web sites are two tactics that may or may not yield Web sites that contain reliable information. This demonstrates a high need for direct teaching of Web literacy skills, and this area should be further explored in the future. STEM in Future Careers Participants were asked to list one job they would like to have as an adult. 70% of participants (N=311) listed careers ranging in the following fields: • Research • Education • Justice/Military • Business • Medicine • Sports • Arts • Specialized services The top five listed careers were: • Teacher • Veterinarian • Lawyer • Doctor • Police Officer For a detailed list of careers mentioned, see Appendix J. Participants were then asked whether they would need to use science, math, or technology in the job(s) they listed. Over 50% of participants (N=238) thought they would (see Figure 20).

Evaluation of MSP2

Finally, participants were asked to list five different jobs they were aware of that involve science, math, or technology. 71% of participants (N=312) listed careers in the above fields—with lots of overlap with specific careers mentioned (see Appendix K for full list). However, the top five careers mentioned in response to this question were: • Teacher • Scientist • Engineer • Doctor • Biologist In general, participants were just as general or specific in naming careers that involve STEM as they are in naming careers in which they are interested, though more generic careers (e.g., doctor, teacher, engineer) were mentioned more frequently than specific careers. Science and Math Topics Participants were asked to list some of the science and math topics they enjoyed learning. While the complete list is broken into science and math in Appendix L, the top five science and math topics mentioned are listed as follows:

Evaluation of MSP2

• • • • •

Science Life science topics in general (N=54) Chemistry/chemicals (N=49) Human body/anatomy/body systems (N=48) Animals/zoology (N=46) Volcanoes (N=36)

• • • • •

Math Arithmetic (N=178) Fractions (N=115) Algebra topics in general (N=74) Geometry topics in general (N=58) None (N=40)

Human body/anatomy/body systems seems to be the singular science topic that both youth enjoy learning and educators often use to teach science concepts; fractions seems to be the singular math topic that both youth enjoy learning and educators often use to teach math concepts. In addition, it seems that youth are particularly drawn to life science and earth science concepts. Youth are also drawn to general topics in algebra and geometry, but most would rather enjoy more elementary math concepts, such as arithmetic. It should also be noted that a large number of youth were not able to think of a topic they actually enjoyed. Youth STEM Exposure Outside of School Participants were asked about their participation in STEM activities outside of school. Figure 21 demonstrates that a majority of participants who answered this item did participate in at least one or two STEM activities outside of school per year.

Participants were also asked to identify specific places where they participated in STEM activities outside of school. Large numbers of participants identified museums, science centers, and planetariums. The next largest type of participation occurred during visits to colleges and university. Small percentages of participants identified STEM participation outside of such established formal and informal learning institutions, these places include neighborhoods and businesses. Participants seem to be able to more readily identify a STEM learning experience when at an institution where they expect it; it is less likely that a STEM learning experience will be identified in more everyday contexts. See Table 5 for specific types of STEM participation.

Evaluation of MSP2

Table 6. STEM Participation Outside of School, by Type Type of Participation N Museum (Art, History) 235 Science museum/Science center/Planetarium 146 College/University 94 Zoos/Aquariums 36 Neighborhood locations (neighbor's, community center, animal shelter, blood bank, environmental center, fire house, food bank, nursing home, SPCA, 19 airport, fairs) Community service (picking up litter, helping kids read, Make-A-Wish project, walks, TailWaggers911) 13 National/State park/Wildlife park 9 Businesses (catering weddings, Coca-Cola Factory, Crayola Factory, Apple Store, IMAX, mall, drug store, pretzel factory) 8 Camp (Space, Little Buffalo, iCamp, Science, 8 Zoology) Historic places (FDR's house, King's Landing, Monticello, Morse's home, Ft. Williams, Pearl Harbor, Gettysberg 8 Military Base/NASA 8 Community organizations (Boy Scouts, Builders Club, Girl Scouts, JMG, Project Reach, Tech Student Association, youth group, Yearbook Club, book club, Red Cross) Activities (cooking, planting flowers, hiking through woods) Religious organization Major city (NYC, DC, Philadelphia) Arts (general participation, backstage, sound board) Science fair/Math bowl Class (First Aid, game design, science workshop) Government Homework/Tutoring Job/Parent's job Library Sports Missing

%* 53.41% 33.18% 21.36% 8.18%

4.32% 2.95% 2.05%

1.82% 1.82%

1.82%

6 6 5 4 4 3 3 3 3 3 2 213

1.36% 1.36% 1.14% 0.91% 0.91% 0.68% 0.68% 0.68% 0.68% 0.68% 0.45% 48.41%

*Total exceeds 100% as respondents were able to indicate multiple types of participation.

Evaluation of MSP2

Resource Title

Resource Guide

references NCTM communicati on standard

references NCTM assessment standards

Alignment with standards

relevant to middle school math teachers and their nonmath colleagues

relevant to middle school math teachers

Relevance

the resource is adaptable but more details should be offered to math teacher to help them work alongside their nonmath colleagues to integrate QL into their content areas

is adaptable

Adaptability

no apparent inaccuracies

Accuracy

A timely and up to date

timely and up to date

Currency

“References” section offers readings and research

readings include timely issues regarding assessment

Research/ Readings

MSP2 Resource Guide and Blog Review (Spring 2010)

QL across the curriculum needs to become as widespread as reading across the curriculum. How can you build upon or off of this resource guide to promote integration of QL into the entire middle school curriculum?

Comments:

May want to include a research section that offers evidence of the benefits of a well balanced assessment system

Comments:

Blog

wide range of resources

includes a broad array of the nuances of assessment; may want to consider organizing strand specific assessment resources A

Coverage

easy to navigate

Navigability

Appeal

good appeal

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Quantitative Literacy http://wiki.nsdl.org/index.php/Mid dleSchoolPortal/Quantitative_Lit eracy

Math Assessment http://wiki.nsdl.org/index.php/Mid dleSchoolPortal/Math_Assessm ent

Appendix K

Evaluation of MSP2

You may want to add subtitles containing math content connections for each of the pieces. Teachers might be more apt to explore them.

Comments:

May want to further investigate some of the upper elementary ELA techniques and strategies and directly relate them to middle grades mathematics.

Comments:

Resource Guide

Connects to NCTM connections and problem solving standards

reference to NCTM PSSM

Alignment with standards

While much of the content is relevant, I wonder if teachers will incorporate it in their planning let alone peruse it if they do not see explicit math content connections

Very relevantespecially to those teachers in a middle school where content tends to be more integrated.

Relevance

While the problems are relevant, determining where they might fit into a teacher’s curriculum may be a challenge for some teachers.

Many sources of adaptable ideas.

Adaptability

no apparent inaccuracies

Accuracy

Seems inapplicable here.

While readings are consistent highlighting the benefits of reading/writin g in mathematics, this piece is missing specific research findings that advocate their inclusion. This might be a selling point for some resistant teachers. D

Research/ Readings

Given the stress in the financial markets this seems timely.

very timely given the push for improvement in language literacy

Currency

Broad array of resources.

offers a wide selection of reading & writing resources

Coverage

easy to navigate

Navigability

Appeal

good appeal

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Money, Money, Money http://wiki.nsdl.org/index.php/Mid dleSchoolPortal/Money_Money_ Money

Reading & Writing Math http://wiki.nsdl.org/index.php/Mid dleSchoolPortal/Reading_and_ Writing_Mathematics

Resource Title

Blog

Comments

Standards mentioned in passing.

Relevant to middle school population

Seems more tangential to the standards than other resource guides

Accuracy

No apparent inaccuracies

Adaptability

Resources are adaptable

Adaptability

Relevance

Alignment with standards

Relevance

Currency

The need for community and family engagement is continual.

Currency

Research/ Readings

Seems inapplicable here.

Research/ Readings

Coverage

Broad array of resources.

Coverage

Navigability

easy to navigate

Navigability

Appeal

B Especially teachers engaged in textbook oriented instruct-‐ tional practices.

Appeal

good appeal

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Quantitative Literacy

Comments:

Resource Guide

Resource Title

Math Fairs, Family Nights, etc. http://wiki.nsdl.org/index.php/Mid dleSchoolPortal/Math_Fairs_and _Competitions

Resource Title

Alignment with standards

Resource Guide

Blog Blog

Evaluation of MSP2

x Comments

B The two sources that are cited (AAAS, NSES) both go to some extent to stress the need to engage students in the Nature of Science prior to the focus on developing vocabulary skills. I found the focus on strategies directed towards amending the deficit in the deficit model of education to be significant. The author’s use of textbooks providing new vocabulary words that are “new” denies the un-‐ readable nature of many textbooks for students that are classified as being below grade level

B Urban strategies where connection s are made to the students lives is a missing component

B Approach is limited to the delivery of informatio n to the student rather than the student engaging the material.

B B This See approach Research supports traditional approache s to teaching science in the classroom. Nature of Science component are absent

B See Research

The publisher will obviously promote their product.

Appears to endorse one publisher over another.

I do not think that it is appropriate to link vocabulary developme nt to a publishers web-site.

B Students that are motivated to learn seem to be targeted for this material.

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Vocabulary Development (cont.)

Vocabulary Development

Comments A very nice collection of resources.

Comments

Alignment with standards

Nice description

Alignment with standards

Adaptability

Relevance

Adaptability

Relevance

The focus seems to be on the student working with material that the teacher provides rather than encouraging students to engage in their self selected (more relevant) texts.

Accuracy

Many of the resources are designed for teacher directed instruction. While the students are engaged in “hands-‐on” activities, they are disproportion ately teacher directed.

Research/ Readings

Currency

Coverage

Going to McGregors webinars takes me to: www.heinema nn.com.com

Navigability

Links to “Crime Scene Online” require log-in. This may be posted on the main page

Fly links do not work

Navigability

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Reading Comprehension Strategies in Science

Resource Title

Forensic Science

Resource Title R e s o u r c e B G l u o i g d e R e s o u r c e B G l u o i g d e

Evaluation of MSP2

Appeal

Comments

Alignment with standards

Comments

A The resources here will easily be adaptable to enable students to apply and relate this important topic to their lives. Critical to this investigation is making that personal connection.

A Nice accompanying discussion of standards and their interpretation.

Relevance

While a large percentage of the U.S. population lives in coastal areas, millions do not. Connecting ocean themes to urban Mid-‐ westerners is very abstract. The web-‐site needs to connect oceans to the lives of these students. Eg. Winter snows of 2010 to warming oceans…

Relevance

Alignment with standards

Adaptability

See Relevance

Adaptability

Accuracy

Research/ Readings

Currency

Coverage

See relevance

Coverage

link not found.

Navigability

Heredity

Navigability

A/B

Appeal

Site designed to be used by both teachers and students who can connect to the ocean experience. Great potential for independent work on the students’ parts.

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Reproduction and Heredity

Resource Title

Ocean Systems

Resource Title R e s o u r c e B G l u o i g d e R e s o u r c e B G l u o i g d e

Evaluation of MSP2

Comments

Alignment with standards

Relevance

Connecting the ocean to the Midwest is a difficult task but the resources here could facilitate this process. Making these connections to the daily lived experience is the key. Maybe a separate section specifically pointing to Midwest issues would assist.

A NSES

Relevance

Alignment with standards

The links for stimulating science fair projects are very good with the exception of “Science Fair Activities from Energy Kids”. This site is a classic what not to do with science fair…cook book experiments…t he kids will have very little engagement with them.

Adaptability

Accuracy

Research/ Readings

Currency

Coverage

Navigability

Formation of Tsunamis Link does not link… Several others do not function as well.

Navigability

Appeal

Almost makes me want to return to my classroom and launch a science fair next year.

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Science Fair 2.0

Resource Title

Earth’s Oceans

Resource Title R e s o u r c e B G l u o i g d e R e s o u r c e B G l u o i g d e

Evaluation of MSP2

Resource Title

Resource Guide Comments:

Blog Relevant to the NCTM representatio n process standard

Alignment with standards Middle school students are use to many visual images.

Relevance

Resources are adaptable

Adaptability

No apparent inaccuracies

Accuracy

none mentioned, may want to share research that shows the benefits of visual thinking in mathematics

Research/ Readings

MSP2 Resource Guide and Blog Review (Spring 2010)

makes meaningful use of technology advances

Currency

might want to acknowledge other math areas in the blog post where readers might contribute visualization tools. while the intent of the post wasn’t to be comprehensi ve, it should include details about future areas to explore

Coverage

easy to navigate

Navigability

Appeal

good appeal

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Making Math Visual http://expertvoices.nsdl.org/middl e-school-mathscience/2008/10/20/makingmath-visual/

Appendix L

Evaluation of MSP2

Comments:

This post was difficult to evaluate from a mathematics lens. Stronger connections to the embedded mathematics needs to be made.

Comments:

Resource Guide

not mentioned, each of the web resources could be identified with a math strand to help organize the content

not mentioned

Alignment with standards

relevance is apparent insofar as potential middle school student engagement

case needs to be made how and why to connect to mathematics

Relevance

adaptable across grade settings and learning settings (i.e. classroom, lunch & learn, after school)

not evaluated for adaptability

Adaptability

no apparent inaccuracies

not evaluated for accuracy

Accuracy

none mentioned, may want to share research that shows the benefit of game; or further resources regarding math games

none mentioned

Research/ Readings

student engagement via gaming, particularly with technology, seems to be increasing

the overarching topic seems current given the national conversation s about health care

Currency

the post leaves me wondering about the selection of online resourceswhy were these ones chosen and other ones not chosen. As with the visualization post, I think that a framework based on content would help show how these resources fit within the framework as well as suggest gaps of resources.

not evaluated

Coverage

lots of spam type comments. may want to consider appending the blog post with user comments that are helpful in expanding the message of the blog post

easy to navigate

Navigability

Appeal

good appeal

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Let’s Talk Teaching: Games in Math Class http://expertvoices.nsdl.org/middl e-school-mathscience/2008/10/27/lets-talkteaching-games-in-math-class/

Tracking Nutrition and Fitness Goals http://expertvoices.nsdl.org/middl e-school-mathscience/2008/10/23/trackingnutrition-and-fitness-goals/

Resource Title

Blog

Evaluation of MSP2

100

As the blog posts expand in topic and frequency it would be a good idea to connect to previous/future posts via links. For example, this post should be linked to the post titled “Online Math Lessons for the Offline Classroom”

Comments:

Consider differentiating resources for individual classroom use and those resources for professional collaboration or discussion

Comments:

Resource Guide

Aligns with NCTM’s PSSM technology principle

aligns with NCTM’s PSSM assessment principle

Alignment with standards

appropriate

ongoing relevance

Relevance

adaptable across settings and grades

Adaptability

no apparent inaccuracies

Accuracy

none offered

the majority of the resources mentioned tend to come from authorities in mathematics education

Research/ Readings

links work and are current

Unfortunately blog posts become outdated as soon as they are published. in particular, this post mentions the 2005 NAEP, yet since then data has been released for 2007 and 2009. A

Currency

see comments

While this post offers a wide collection of items, I’m left wondering why these resources were selected.

A-B

Coverage

easy to navigate

The “Math Whizz” spam should have not been allowed to be posted.

Navigability

Appeal

good appeal

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Exploring Math Interactivity - in Your Offline Classroom http://expertvoices.nsdl.org/middl e-school-mathscience/2009/03/13/exploringmath-interactively-in-your-offlineclassroom/

Re-thinking Math Assessment http://expertvoices.nsdl.org/middl e-school-mathscience/2008/12/18/re-thinkingmath-assessment/

Resource Title

Blog

Evaluation of MSP2

101

The text “If students can learn to explain their thinking in solving a math problem (using drawings or tables or graphs as well as words), they acquire a means of setting out their work logically and refining their thinking as they communicate their understandings” suggests a connection to the standard of representation.

From standards…

Comments:

A recurring thought as I read through the blog posts is “what could be missing”. In this post, no resources consider measuring a sphere, or even a frustum for example. I wonder if the conclusion of a post mentioned the gaps still needing resources might encourage reader participation while filling in the gaps of resources. X

Comments:

Resource Guide

Connected to NCTM PSSM Communicati on standard. (continued in comments section)

This aligns with the NCTM PSSM measuremen t standard. It might be helpful to explicitly state this and hotlink to the NCTM standard.

A-B

Alignment with standards

relevantespecially in a school setting that is a middle school where crosscurriculum integration occurs

relevant

Relevance

seems adaptable

Adaptability

no apparent inaccuracies

Accuracy

none offered

Research/ Readings

links work and are current

Currency

see comments

Coverage

easy to navigate

Navigability

Appeal

As mentioned under alignment- a focus on the standard representatio n may be a blog or wiki topic worth exploring.

This post primarily serves as an advert for the math resource guide “Reading and Writing Math”.

good appeal

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Reading and Writing Mathematics http://expertvoices.nsdl.org/middl e-school-mathscience/2009/06/29/reading-andwriting-mathematics/

Measuring a Solid http://expertvoices.nsdl.org/middl e-school-mathscience/2009/04/15/measuringa-solid/

Resource Title

Blog

102

Comments

NSES generalities

Alignment with standards

Relating the issues of the polar bear to local experiences… e.g. if the bears are experiencing these changes, what changes can we find evidence for locally.

Relevance

Relevant material connected to the controversy with this topic would be helpful.

Relevance

Alignment with standards

Adaptability

A lack of response to the blogger who commented about the cold winter in OH indicating that global warming could therefore not be happening; this is typical lack of understanding of the issue. Immediate graph and NOAA link of averages should be posted in response.

Accuracy

As the blog indicates, much more needs to be added here. Research (Colburn & Henriques 2006) indicates that clergy is more apt than teachers to correctly understand founding principles of evolution. Teachers need OVERT support along these lines. Many may not know what questions they need to be asking.

Accuracy

Research/ Readings

Currency

Let us not forget the controversy that this topic presents.

Currency

Coverage

Connections to local relevance

Coverage

Navigability

Link to “Heredity” produced “page not found Locating the links at the top of the page could assist browsing later.

Navigability

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Polar Bears and Climate Change

Resource Title

Resource Title The Origin of Species R e s o u r c e B G l u o i g d e R e s o u r c e B G l u o i g d e

Evaluation of MSP2

Appeal

Comments

Alignment with standards

Relevance

Alignment with standards

Excellent venue for connecting to individual and local concerns.

Adaptability

Nice suggestions for inquiry extensions

Adaptability

Accuracy

If teachers do not understand the science behind the concept…how will they assist the students in grasping the concepts?

Research/ Readings

Inquiry suggested as extension but not as a primary focus.

Research/ Readings

Currency

Coverage

Suggesting to teachers relevance to different cell functions and their relative sizes.

Coverage

Navigability

Appeal

Links support interdisciplina ry work.

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

How Does Bird Diversity Affect the Incidence of West Nile Virus in Humans?

Resource Title

Investigating Cell Size

Resource Title R e s o u r c e B G l u o i g d e R e s o u r c e B G l u o i g d e

Evaluation of MSP2

103

Comments

Alignment with standards

Relevance

Alignment with standards

Adaptability

A significant portion of the blog relates to the inquiry methods as it relates to NOS. The literature on NOS is loaded with findings that clearly state that a majority of teachers of science do not have an understanding of NOS and therefore have tremendous difficulty teaching with inquiry. This posting needs to be much more explicit in what it means by being “explicit” when teaching inquiry/NOS.

Adaptability

Accuracy

See adapatbility

Accuracy

Research/ Readings

Semantics are at play here. “Commonly accepted” research findings do not necessarily reflect current research. Developing NOS in students will require a dramatic change in classroom practices. So, while the content in this blog is timely and current, it seems to ignore much of current practice. A synopsis of current NOS literature may be helpful here.

Research/ Readings

Currency

Downgraded only due to the recent earthquake news omitted due to the lack of updating of site. However, the links are fantastic.

Currency

Coverage

See adapatbility

Coverage

Navigability

Adapted from The Ohio Resource Center’s: ORC Resource Review Rubric retrieved March 2, 2010 from: http://www.ohiorc.org/about/rubric.aspx.

Middle Level Students and ‘Abilities Necessary to Do Scientific Inquiry’

Resource Title

Are You Ready for Richter Scale Day?

Resource Title R e s o u r c e B G l u o i g d e R e s o u r c e B G l u o i g d e

Evaluation of MSP2

104

Appeal

Appendix M

7/29/09 What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

25% (1) 0% (0) 0% (0) 25% (1)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

0% (0) 50% (2) 0% (0)

0% (0) 0% (0) 50% (2)

2 - Middle Grades (5-8) 4 - School District

50% (2) 0% (0)

Mean: 4 Response: 4

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other Mean: 3.5 Response: 4

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

100% (4) 50% (2)

2 - Presentation 4 - Other

75% (3) 25% (1)

Response: 4

Evaluation of MSP2

105

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 100% (4)

4 0% (0)

3 0% (0)

2 0% (0)

1 0% (0)

Mean 1

Response: 4

What was the biggest takeaway for you from this session? 1 2 3 4

See what can be done with Moodle and students. examples of ways to use interactive notebooks and to manage them when you have lots of students; would love to have a followup session focusing on more of the tech tools you could use for interactive notebooks with pros/cons for the various tools. The discussions and the examples plus where to go for more information Setting up the notebook for success.

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

25% (1) 25% (1)

2 - No

50% (2)

Mean: 2 Response: 4

What other topics would you like to see for future MSP2 Tech Talks or online events? 1 2

Using Notebook Software on interactive whiteboards, Twitter in the classroom, Voicethreads Thank you for providing this session. looking forward to the next topics you have planned: Moodle, wikis, digital storytelling (don't see a comment box for requesting additional information about MSP2 Tech Talks but I am interested in learning more--learned about this session on LearnCentral) I have the link for the MSP2 portal and I'll do some exploring there. Thanks!

www.clicktools.com

Evaluation of MSP2

106

1 - MSP2 Tech Talk Information

100% (5)

Response: 5

1 - Elluminate product information

100% (4)

Response: 4

First Name

Last Name

Joe

Testcase

Joe

Testcase

Kathleen

Burgess

Peggy

George

Cribb

Organizatio n

Title

State/Provin Country Email Phone ( Email Phone ( ce (Required (Required ex. 123-456- (Required ex. 123-456(Required for Contact) for Contact) 7890) for Contact) 7890) for US or Canada) Test joe@test.co m Test joe@testme. com Archdiocese Library/Medi Pennsylvani US djburgess1 610-532djburgess1 610-532of a a @aol.com 3335 @aol.com 3335 Philadelphia Curriculum Chairperson AZ USA pgeorge@m ac.com Tabor city Media NC US jlcribb@colu 910-653cribbjl@yah 910-653Middle Specialist mbus.k12.n 3637 oo.com 3715 School c.us Generated using clicktools on Tuesday October 27 2009 14:18:52

www.clicktools.com

Evaluation of MSP2

107

National Middle School Association What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

72.73% (8) 0% (0) 0% (0) 0% (0)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

0% (0) 27.27% (3) 0% (0)

18.18% (2) 9.09% (1) 9.09% (1)

2 - Middle Grades (5-8) 4 - School District

63.64% (7) 0% (0)

Mean: 1.82 Response: 11

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other Mean: 2.18 Response: 11

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

72.73% (8) 27.27% (3)

2 - Presentation 4 - Other

27.27% (3) 18.18% (2)

Response: 11

Evaluation of MSP2

108

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 54.55% (6)

4 36.36% (4)

3 9.09% (1)

2 0% (0)

1 0% (0)

Mean 1.55

Response: 11

What was the biggest takeaway for you from this session? 1 2 3 4 5 6 7 8 9

I crashed the moodle site for my district yesterday. They said I had too much on it. This sounded like I can explore some other ways of doing things. all I have more confidence to try Moodle. I am also hopeful that I can continue this dialogue at MSP2 with other interested teachers! I am moving to the 6-9 grade level and am encouraged by your success using Moodle with those students. links to browse and go to for help Examples that any teacher could use, no matter what the platform. What a moodle site can do and look like I found a lot of new information. tips and resources shared

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

9.09% (1) 0% (0)

2 - No

90.91% (10)

Mean: 1.91 Response: 11

What other topics would you like to see for future MSP2 Tech Talks or online events? 1 2 3 4 5

some example Maths course Noteshare program or webpage use in the classroom Anytime you can get an exemplary teacher to share, it's great. My system isn't even using Moodle, but I still got great tips. follow up to today's moodle web 2.0 in the classroom real examples

www.clicktools.com

Evaluation of MSP2

109

1 - MSP2 Tech Talk Information

100% (5)

Response: 5

1 - Elluminate product information

100% (4)

Response: 4

First Name

Last Name

Organizatio n

Title

Ann

Meyer

LMS and ITL

virginia

alberti

George

Mayfield

Fairview Charter School ITIS and LST B.Castelli GOAL Academy

Henry

Hill

Susie

Highley

Marie

Clarner

Celena

Miller

Creston Middle School North Attleboro Middle School

teacher

Instructor

State/Provin Country Email Phone ( Email Phone ( ce (Required (Required ex. 123-456- (Required ex. 123-456(Required for Contact) for Contact) 7890) for Contact) 7890) for US or Canada) Wisconsin United meyeral@mi 414-546- mathnut@wi 414-327States lwaukee.k12 7707 .rr.com 2391 .wi.us ITALY viralber@tis 0039030375 cali.it 2984 Colorado

USA

Florida Media Specialist

United States USA

Math Teacher

USA

geomay47 @gmail.com henrythill@g mail.com shighley@a ol.com

719-2423697

mclarner@n aschools.net

508-643230X4126

mclarner@n 508-643aschools.net 2130X4126

celena_mille r@hotmail.c om Generated using clicktools on Tuesday October 27 2009 14:20:01

www.clicktools.com

Evaluation of MSP2

110

August 12, 2009 What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

57.14% (4) 0% (0) 0% (0) 28.57% (2)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

0% (0) 0% (0) 14.29% (1)

0% (0) 0% (0) 71.43% (5)

2 - Middle Grades (5-8) 4 - School District

28.57% (2) 0% (0)

Mean: 3.43 Response: 7

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other Mean: 4.14 Response: 7

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

71.43% (5) 42.86% (3)

2 - Presentation 4 - Other

57.14% (4) 28.57% (2)

Response: 7

Evaluation of MSP2

111

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 71.43% (5)

4 14.29% (1)

3 14.29% (1)

2 0% (0)

1 0% (0)

Mean 1.43

Response: 7

What was the biggest takeaway for you from this session? 1 2 3 4 5 6 7

links for teacher support 7 principles of storytelling, what people do with stories, particular math/science "quirks" of stories Start up tips Practical ideas steps in developing digital stories and examples--looking forward to exploring the resources on the Ning later great ideas shared! The slide information and links shared.

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

0% (0) 0% (0)

2 - No

100% (7)

Mean: 2 Response: 7

What other topics would you like to see for future MSP2 Tech Talks or online events? 1 2 3 4

Social Media in Math (Math 2.0) Further opportunities to share struggles and successes with Digital Storytelling once participants have created own story. love the topics you have chosen! looking forward to the wiki session! how about an entire session on Voicethread--awesome, flexible tool for many things and especially digital storytelling and collaborative student learning Digital storytelling tools to work with visually handicapped students.

www.clicktools.com

Evaluation of MSP2

112

1 - MSP2 Tech Talk Information

100% (4)

Response: 4

1 - Elluminate product information

0% (0)

Response: 0

First Name

Last Name

Maria

Droujkova

Pamela

Burish

Basilla

Stevens

Peggy

George

Organizatio n

Title

State/Provin Country Email Phone ( Email Phone ( ce (Required (Required ex. 123-456- (Required ex. 123-456(Required for Contact) for Contact) 7890) for Contact) 7890) for US or Canada) Natural Director NC USA droujkova@ Math gmail.com University of Coordinator IN United pjburish@us 574-631- pburish@nd Notre Dame for States a.net 0193 .edu Technology Outreach/Fa culty Seymour Teacher Connecticut USA bstevens10 2038887690 Middle 10@snet.ne School t AzTEA AZ USA pgeorge@m ac.com Generated using clicktools on Tuesday October 27 2009 14:21:37

www.clicktools.com

Evaluation of MSP2

113

9/16/09 What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

25% (1) 0% (0) 0% (0) 25% (1)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

0% (0) 50% (2) 0% (0)

25% (1) 0% (0) 25% (1)

2 - Middle Grades (5-8) 4 - School District

25% (1) 25% (1)

Mean: 4 Response: 4

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other Mean: 3 Response: 4

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

100% (3) 33.33% (1)

2 - Presentation 4 - Other

66.67% (2) 0% (0)

Response: 3

Evaluation of MSP2

114

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 33.33% (1)

4 66.67% (2)

3 0% (0)

2 0% (0)

1 0% (0)

Mean 1.67

Response: 3

What was the biggest takeaway for you from this session? 1 2 3

I came in at the end--hope to see the recording. Glad to see there are others out there who think the same as we do. Great resources. The suggestions of sites to visit for more information

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

0% (0) 25% (1)

2 - No

75% (3)

Mean: 2.25 Response: 4

What other topics would you like to see for future MSP2 Tech Talks or online events? 1

More on using Google Apps in the classroom for interactive student learning

1 - MSP2 Tech Talk Information

0% (0)

Response: 0

www.clicktools.com

Evaluation of MSP2

115

1 - Elluminate product information

0% (0)

Response: 0

First Name

Last Name

Organizatio n

Title

State/Provin Country Email Phone ( Email Phone ( ce (Required (Required ex. 123-456- (Required ex. 123-456(Required for Contact) for Contact) 7890) for Contact) 7890) for US or Canada)

Generated using clicktools on Tuesday October 27 2009 14:23:05

www.clicktools.com

Evaluation of MSP2

116

10/7/09 What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

0% (0) 0% (0) 0% (0) 33.33% (2)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

0% (0) 66.67% (4) 0% (0)

0% (0) 16.67% (1) 66.67% (4)

2 - Middle Grades (5-8) 4 - School District

16.67% (1) 0% (0)

Mean: 5 Response: 6

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other Mean: 4.17 Response: 6

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

66.67% (4) 50% (3)

2 - Presentation 4 - Other

100% (6) 0% (0)

Response: 6

Evaluation of MSP2

117

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 83.33% (5)

4 16.67% (1)

3 0% (0)

2 0% (0)

1 0% (0)

Mean 1.17

Response: 6

What was the biggest takeaway for you from this session? 1 2 3 4 5 6

Just needed a refresher on how to use Diigo and the differences btw that and delicious which I already use. Show students how to use Diigo for research. Diigo possibilities A Diigo account is in my near future. how to use diigo, comparison to delicious, great diigo tools Diigo is a powerful social bookmarking/research tool for teachers and students!

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

0% (0) 0% (0)

2 - No

100% (6)

Mean: 2 Response: 6

What other topics would you like to see for future MSP2 Tech Talks or online events? 1 2

Edmodo Would love to see a followup session on Diigo with examples of how teachers are using it with students in their classrooms

www.clicktools.com

Evaluation of MSP2

118

1 - MSP2 Tech Talk Information

100% (1)

Response: 1

1 - Elluminate product information

0% (0)

Response: 0

First Name

Last Name

Organizatio n

Geordie

Paulus

Alice

Tope

Elkridge Landing Middle etseo

Title

Generated using clicktools on Tuesday October 27 2009 14:24:24

www.clicktools.com

Evaluation of MSP2

119

10/21/09 What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

16.67% (1) 0% (0) 0% (0) 33.33% (2)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

0% (0) 16.67% (1) 66.67% (4)

2 - Middle Grades (5-8) 4 - School District

50% (3) 0% (0) 0% (0)

Mean: 3.5 Response: 6

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other

16.67% (1) 0% (0)

Mean: 4.17 Response: 6

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

66.67% (4) 0% (0)

2 - Presentation 4 - Other

50% (3) 0% (0)

Response: 6

Evaluation of MSP2

120

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 50% (3)

4 50% (3)

3 0% (0)

2 0% (0)

1 0% (0)

Mean 1.5

Response: 6

What was the biggest takeaway for you from this session? 1 2 3 4

CAST has lots of free tools that I didn't know about - tools that will help me integrate UDL best practices into my work three components of UDL, Book builder, and UDL sample courses for teacher prep Definition of UDL and sites where I can go for help on this topic. Ideas about where to goe for more infor about UDL

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

33.33% (2) 16.67% (1)

2 - No

50% (3)

Mean: 1.83 Response: 6

What other topics would you like to see for future MSP2 Tech Talks or online events? 1

algebra software

www.clicktools.com

Evaluation of MSP2

121

1 - MSP2 Tech Talk Information

100% (1)

Response: 1

1 - Elluminate product information

0% (0)

Response: 0

First Name

Last Name

Karla

Phillips

Organizatio n

Title

State/Provin Country Email Phone ( Email Phone ( ce (Required (Required ex. 123-456- (Required ex. 123-456(Required for Contact) for Contact) 7890) for Contact) 7890) for US or Canada) Coconino Curriculum AZ USA karla.phillips 928-226- karla.phillips 928-226Community Coordinator @coconino. 4346 @coconino. 4346 College edu edu Generated using clicktools on Tuesday October 27 2009 14:53:59

www.clicktools.com

Evaluation of MSP2

122

November 2009 What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

50% (3) 0% (0) 33.33% (2) 16.67% (1)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

16.67% (1) 0% (0) 16.67% (1)

2 - Middle Grades (5-8) 4 - School District

0% (0) 0% (0) 0% (0)

Mean: 3.33 Response: 6

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other

66.67% (4) 0% (0)

Mean: 2.33 Response: 6

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

83.33% (5) 16.67% (1)

2 - Presentation 4 - Other

100% (6) 0% (0)

Response: 6

Evaluation of MSP2

123

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 50% (3)

4 50% (3)

3 0% (0)

2 0% (0)

1 0% (0)

Mean 1.5

Response: 6

What was the biggest takeaway for you from this session? 1 2 3 4

All the safety proceedures I did not know about in the science classroom. I found all of the information really useful. I appreciated the pictures because it will make it easier for me to literally see hazards in the classroom. How many things are done incorrectly in the lab setting in schools. Dr. Roy's dialogue about common misconceptions about laboratory safety.

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

33.33% (2) 33.33% (2)

2 - No

33.33% (2)

Mean: 2 Response: 6

What other topics would you like to see for future MSP2 Tech Talks or online events? 1 2 3

Using technology in the classroom examples of lessons and hands-on activities to use in the classroom Something on integrating Project Wild or Project Wet into state science standards, especially when working through a curriculum guide.

www.clicktools.com

Evaluation of MSP2

124

1 - MSP2 Tech Talk Information

0% (0)

Response: 0

1 - Elluminate product information

0% (0)

Response: 0

First Name

Last Name

Organizatio n

Title

State/Provin Country Email Phone ( Email Phone ( ce (Required (Required ex. 123-456- (Required ex. 123-456(Required for Contact) for Contact) 7890) for Contact) 7890) for US or Canada)

Generated using clicktools on Monday April 19 2010 20:16:50

www.clicktools.com

Evaluation of MSP2

125

National Middle School Association What is your primary role as an educator?

1 - Teacher/ Classroom Educator 3 - Curriculum Developer 5 - Student 7 - Other

40% (8) 5% (1) 0% (0) 15% (3)

2 - Administrator 4 - Library, Resource, or Technology Coordinator 6 - Researcher

15% (3) 25% (5) 0% (0)

2 - Middle Grades (5-8) 4 - School District

15% (3) 15% (3)

Mean: 2.9 Response: 20

In what setting do you primarily work?

1 - Elementary Grades (preK-4) 3 - Secondary Grades (9-12) 5 - Other

5% (1) 15% (3) 50% (10)

Mean: 3.9 Response: 20

What has been the most beneficial aspect of today's session?

1 - Examples and Illustrations 3-Q&A

65% (13) 5% (1)

2 - Presentation 4 - Other

50% (10) 10% (2)

Response: 20

Evaluation of MSP2

126

(5=excellent, 4=very good, 3=good, 2=adequate and 1=poor)

Rate the quality of the session content

5 57.89% (11)

4 36.84% (7)

3 5.26% (1)

2 0% (0)

1 0% (0)

Mean 1.47

Response: 19

What was the biggest takeaway for you from this session? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

site examples Overview of Scratch ways to use tools to teach math and which tools to use Use of a variety of apps. to engage students in a math classroom. different tools More ways that teachers can benefit from using these tools. Questions from other participants, how the session was facilitated, explanations of applications I wasn't already familiar with Didn't learn much new, but enjoyed the presentation Just expanded my knowledge of what is available on Google, a great deal lots to explore Google Books what google tools to start with as a new user Very interesting applications I was not aware of. I would like to try and incorporate it in my learning environments. I would also like to try and introduce it to my children's teachers. Knowledge of ways to use various apps Learning how to use all aspects of Google applications. Ideas for how to use Google Apps. Quick overview of what's available and it's use. The overview for GoogleApps for Education

Did you have any technical difficulties during the session?

1 - Yes 3 - If yes, please explain

5% (1) 5% (1)

2 - No

90% (18)

Mean: 2 Response: 20

What other topics would you like to see for future MSP2 Tech Talks or online events? 1 2 3

more web 2.0 tools connecting classrooms across borders Specifics on teaching Algebra using technology.

www.clicktools.com

Evaluation of MSP2

127

4 5 6

math applet math tools about simulations digital storytelling tools/strategies Strategies for implementing Google Apps in the classroom always interested in Web 2.0 tools, good examples of classroom application of tools, ideas for deliver and moels for classroom based technology PD in districts, I haven't yet looked at your website. Plean to go there tonight. I may find some of this there. How to set up Google Sites GoogleApps used in the classroom, example curriculum, lesson plans, best practices

7 8 9

1 - MSP2 Tech Talk Information

100% (6)

Response: 6

1 - Elluminate product information

100% (1)

Response: 1

First Name

Laura

Karen

Pat

Sharon

Melissa

Susan

Leo

Last Name

Organizatio n

Title

State/Provin Country Email Phone ( ce (Required (Required ex. 123-456(Required for Contact) for Contact) 7890) for US or Canada) Silva Azusa Lecturer CA United lasilva@apu 626-815Pacific Curriculum States .edu 5039 University Specialist Justl Spring Ed. Tech TX USA karen.justl@ 713-251Branch ISD Grant springbranc 2405 Facilitator hisd.com Elliott Hewitt's TeacherOntario Canada Creek P.S. librarian Smith Oklahoma USA sndsmith@y ahoo.com Thibodeaux East Classroom Louisiana US mthibodeau 337-328Beauregard Teacher/Ed x@beau.k1 7511 High School Tech 2.la.us Facilitator Miller Pinellas Teacher Florida USA millersus@p County csb.org Schools Willems Singapore willems.leo @gmail.com

Email Phone ( (Required ex. 123-456for Contact) 7890)

lasilva@apu .edu

willems.leo @gmail.com

www.clicktools.com

Evaluation of MSP2

128

626-8155039