!"##$%&'()'*+%,-.$%.'/).0%-12((,'3%(4%$#'50$."%0-' +6'7(#8$%+-(6'9+.2':.20%'3%(4%$#-' November 2011 Dr. Lisa Raphael, Research Specialist lisa.raphael@sedl.org 512-391-6546 Dr. Cheryl Harris, Research Associate cheryl.harris@sedl.org 512-391-6516
Dr. Zena Rudo, Project Director zena.rudo@sedl.org 512-391-6554 Dr. Melissa Dodson, Evaluation Manager melissa.dodson@sedl.org 512-391-6606
:;0%;+09'()'.20'<(%=' In April, 2011, Girlstart contracted with SEDL to complete a variety of evaluation activities including a comparison of the Girlstart afterschool program with other similar programs in the nation. SEDL’s activities for this component of the evaluation were designed to accomplish two primary tasks: 1) Identify a representative sample of STEM afterschool programs that reflect best practice criteria in the afterschool literature and 2) Analyze data collected to identify similarities and differences between Girlstart and the similar programs with respect to best practices. This brief provides a summary of SEDL’s approach to conducting the comparison and an overview of major findings. >0.2(?-')(%'!0,01.+64'7(#8$%+-(6'3%(4%$#-' SEDL identified and reviewed over 100 viable afterschool programs for possible inclusion in a comparison analysis with Girlstart. To conduct the search, SEDL developed characteristics of afterschool programs similar to the Girlstart program and afterschool best practice criteria, as well as a list of key search terms. The characteristics of afterschool programs SEDL developed included: • Program participants –Girls only • Professional development • Grade levels served • STEM services to the public • STEM focus • Other services • Afterschool and/or summer camps The afterschool best practices SEDL developed included elements related to: • Who attends the program • What grade levels the program serves • Types of STEM activities the program provides • Afterschool program’s structure • Other services the program provides • Program’s curriculum • Types of high quality services the program maintains
1
• •
How students are engaged in the program What impact the program has had
SEDL’s Information Associate conducted a search on EBSCO Academic Search Elite and ERIC databases for recent publications using the following key terms: • Afterschool or After-school • Extended Learning • Out of school • OST (abbreviation for out of school time) • Science club • Afterschool programs • STEM OR science, mathematics, technology, engineering SEDL’s Information Associate also scanned Google Scholar using the following search string: “Afterschool programs” “STEM” “Girls.” Additionally, SEDL evaluators searched the National Girls Collaborative Project database http://www.ngcproject.org/index.cfm and the Harvard Family Research Project’s Out-ofSchool Time Program Research and Evaluation Database and Bibliography http://www.hfrp.org/out-of-school-time/ost-database-bibliography. Lastly, SEDL relied on its national work in the field of afterschool and evaluators and contacted colleagues and afterschool programs requesting recommendations for STEM afterschool programs to consider. Based on these searches, SEDL located 13 full-text articles (see references below) and identified approximately 100 potential afterschool programs for comparison to Girlstart. As a result of these searches and initial scan of program information, SEDL selected four afterschool programs (Girls Inc. of San Antonio, Project Exploration, Science Club for Girls, Cool Girls Art and Science Club) for the comparison with Girlstart that met the criteria developed. All four afterschool programs are focused on STEM content, target girls in grades 1-12, and are currently in operation. SEDL also noted eight other programs that had some similarities, but did not meet enough of the criteria developed (See Appendix A for a list of all 12 programs). Over several weeks, SEDL evaluators conducted interviews (see Appendix B for interview protocol) with key program staff for the four selected afterschool programs and Girlstart. SEDL evaluators developed a matrix that compared quality practices of the four selected afterschool programs with Girlstart (see Appendix C for the matrix). SEDL evaluators shared the matrix with program staff from each of the programs and made changes to the matrix based on their feedback. !"##$%&'()'>$@(%'5+6?+64-' In conclusion, Girlstart is one of five afterschool programs of 100 viable afterschool programs in the United States that specifically targets girls in grades K-12 and exclusively focus their afterschool activities on STEM. The features unique to Girlstart are summarized here, as well as the similarities between Girlstart and the four selected programs. 2
!"#$%&"'(%)*+%"($,(-*&.'$#&$/( • Girlstart’s afterschool programs run longer through the year than the other programs. All of Girlstart’s afterschool programs run for 25 weeks. For Project Exploration, three of its afterschool programs run for 25 weeks but its other two programs run for 10-13 weeks. The other afterschool programs run between 8-16 weeks. •
Girlstart has more participating schools (21) in its afterschool programs than the other four comparison programs. Science Club for Girls currently has six participating schools, six community sites, and the other programs have less.
-*&.'$#&$(#)0(#..(1,%&(2,34#&*',)(#1$"&'25,,.(4&,6&#3'/ • Specifically target girls in STEM • Utilize outside STEM resources and/or students participate in • View themselves as “local” STEM activities in the field programs • Provide modifications for diverse • Occur onsite in K-12 schools or learners community sites • Make connections to academics • Meet weekly for a minimum of in enrichment activities one hour • Rely on university partnerships • Use hands-on, interactive activities • Have regular contact with school day staff • Tailor activities to students needs and interests • Host STEM related events for the public • Make real-world connections ( -*&.'$#&$(#)0($5&""(,1($5"(2,34#&*',)(#1$"&'25,,.(4&,6&#3'/( • Target students with low SES • Have regular professional and underrepresented ethnicity development for program staff • Use research-based curriculum • Have regular staff meetings • Clearly state their mission/goals • Collect and monitor participant on their websites data and records • Employ or use youth • Regularly collect data for the workers/interns purpose of program evaluation ( 7,)"(,1($5"(1*8"(#1$"&'25,,.(4&,6&#3'(5#8"/( • Dedicated structured time specifically for homework completion ( 9$5"&(*34,&$#)$(1*)0*)6'/( • None of the afterschool programs examined have school day teachers working as instructors in the program but Girlstart, Project Exploration, and to some extent, Science Club for Girls involve school day teachers to assist in curriculum planning and/or development.
3
• • •
Both Girlstart and Project Exploration have a standards-based curriculum. Science Club for Girls also has standards represented in some of their curriculum. Of the programs examined, Girlstart and Girls Inc. of San Antonio are the only programs with summer camps. Girlstart and Girls Inc. of San Antonio primarily work with Hispanic/Latina girls.
A+#+.$.+(6-' There are several limitations to SEDL’s identification and comparison of afterschool programs. First, the literature search used to identify afterschool programs for comparison to Girlstart resulted in a list of programs that was not exhaustive and based on specific criteria of afterschool programs. There may be other programs that are not readily accessible through the type of search SEDL conducted. Additionally, SEDL’s use of a program’s publications and reports, or other publications to obtain initial information to select other afterschool programs for comparison may have eliminated some programs for which more thorough information would have resulted in its inclusion. Further, data collected for the matrix that shows that the comparison of programs similar to Girlstart were obtained through interviews and are based on program staff perceptions. Perceptions may differ from actual behaviors for many reasons, such as a desire to provide socially acceptable responses or information. Finally, information obtained for the matrix is based on the current semester. Program staff indicated in the interviews that the number of programs changes each semester. B0)0%0610-'' Afterschool Alert Issue Brief (2008). Afterschool Programs: At the STEM of learning. Washington D.C.: Coalition for Science Afterschool. http://www.afterschoolalliance.org/issue_26_STEM.cfm Afterschool Alliance (2011). STEM learning in afterschool: An analysis of impact and outcomes. Washington D.C.: Afterschool Alliance. http://www.afterschoolalliance.org/researchReports.cfm Bloor, A. Krenitsky, L., Wellenstein, M.J. (2008). Work in progress- women in technology: An initiative to reach females in rural Wisconsin. Paper presented at the 2008 38th Annual Frontiers in Education Conference. http://www.computer.org/portal/web/csdl/doi/10.1109/FIE.2007.4418079 Chun, K., & Harris, E. (2011). Research update 5: STEM out-of-school time programs for girls. Cambridge, MA: Harvard Family Research Project. Retrieved from http://www.hfrp.org/publications-resources/browse-our-publications/research-update-5-stem-outof-school-time-programs-for-girls Davis, K. (2007). Meeting women’s and girl’s special needs: “Gender-sentive” environments and the roadblocks women science educators face. Paper presented at the Meeting of the National Association for Research in Science Teaching (70th, Oak Brook, IL, March, 1997). http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_ SearchValue_0=ED406160&ERICExtSearch_SearchType_0=no&accno=ED406160
4
Froschl, M., Sprung, B., Archer, E., & Fancsali, C. (2003). Science, gender, and afterschool: A research agenda. Washington, DC: Educational Equity Concepts and Academy of Educational Development. http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_ SearchValue_0=ED500856&ERICExtSearch_SearchType_0=no&accno=ED500856 Hirsch, B. J. (2011). Learning and development in after-school programs. Phi Delta Kappan: 92, 66-69. http://www.kappanmagazine.org/content/92/5/66.abstract Huebner, T. A. (2009). Encouraging girls to pursue math and science. Educational Leadership, 67(1), 90–91. http://www.ascd.org/publications/educationalleadership/sept09/vol67/num01/Encouraging-Girls-to-Pursue-Math-and-Science.aspx Jafri, J. & Lyon, G. (2010). Project Exploration’s Sisters4Science: Involving Urban Girls of Color in Science Out of School. Afterschool Matters, 11, 15-23. http://www.robertbownefoundation.org/pdf_files/2010_asm_june.pdf Koenig, Kathleen, and Margaret Hanson (2008). Fueling interest in science: an after-school program model that works. Science Scope, v. 32, Dec. 2008: 48-51. http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_ SearchValue_0=EJ824855&ERICExtSearch_SearchType_0=no&accno=EJ824855 Scott, K.A.; Aist, G.; Hood, D.W. (2009) CompuGirls: Designing a culturally relevant technology program. Educational Technology, 49 (6) 34-39. http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_ SearchValue_0=EJ866119&ERICExtSearch_SearchType_0=no&accno=EJ866119 Weston, V.; Bonhivert, A.; Elia, A.; Hsu-Kim, H.; Ybarra, G. (2008). Work in progress: A STEM educational outreach day for young females. Paper presented at the 2008 38th Annual Frontiers in Education Conference. http://www.computer.org/portal/web/csdl/doi/10.1109/FIE.2008.4720570 Young, P. (2011). STEM learning beyond the school day: Initial survey results help determine where we need to go. Report prepared on behalf of the Afterschool Alliance, the National Summer Learning association, and the National AfterSchool Association. http://www.summerlearning.org/?page=stem_resources
5
Appendix A: Selected Afterschool Programs for Comparison to Girlstart
! Science Club for Girls (SCFG) Karen O’Neill koneill@scienceclubforgirls.org 617-391-0361
Girls Only
Grade Levels
K-12
!
STEM
!
Afterschool
Summer Camps *Weeklong program but not considered summer camp in State
!
Website: http://www.scienceclubforgirls.org/ Mission: Our mission is to increase the self-confidence and literacy in science, technology, engineering and mathematics (STEM) of K-12th grade girls belonging to groups that are underrepresented in these fields, through free programs that include hands-on learning, mentorship, and leadership opportunities. Girls work with mentor-scientists who model and foster leadership, affirm college as an expectation, and promote careers in science and technology as goals and options. Project Exploration's Sisters4Science (S4S)
Staff PD
STEM Conf
Public STEM
!
Catalyst Award Ceremony and Benefit
Show Me the Science (reverse science fair)
Virtual STEM
Other
School vacation week program
Description: SCFG provides free, hands-on science and engineering programs to over 1,000 girls in five cities in eastern Massachusetts (Cambridge, Lawrence, Boston, Newton and Fitchburg), and in Pokuase, Ghana.
Girls Only
Grade Levels
STEM
Afterschool
Summer Camps
Staff PD
!
6-8
!
!
!
!
STEM Conf
Public STEM
Virtual STEM
Other
Kathleen St. Louis kstlouis@projectexploration.org 773-834-7614 Website: http://www.projectexploration.org/servicesfor-girls.php Mission: • Create a safe space for girls to explore science and develop leadership skills. • Expose girls to the wide variety of roles played by women in science. • Improve girls’ overall attitudes about science by developing team building, communication, and
!
Description: Sisters4Science is a year-round after-school and field program that combines science exploration with leadership development for approximately 100 minority middle school girls. Currently, the program is run in partnership with five schools across Chicago. Girls participate in hands-on science activities led by women scientists as well as participate in science-based field trips. The interests of the girls drive the curriculum; in the past, participants have investigated topics ranging from genetics to chemistry to girls’ health. Sessions are held weekly at each school site.
Appendix A: Selected Afterschool Programs for Comparison to Girlstart
! leadership skills. Girls, Inc. of San Antonio: Build It Zelina Cruz, Program Director zcruz@chshel.org 210-212-2517 Website: http://www. girlsincsa.com/ Mission: To â&#x20AC;&#x153;Inspire all girls to be strong, smart and boldâ&#x20AC;?
Cool Girls Science & Art Club
Girls Only
Grade Levels
STEM
Afterschool
Summer Camps
Staff PD
STEM Conf
Public STEM
!
1-5
!
!
!
!
Annual Science Festival
!
Virtual STEM
Other
Holiday Camp
Description: Girls Inc. of San Antonio offers center-based programs like Girls University, an award winning summer camp, and holiday camps throughout the year along with mother/daughter workshops. Outreach programs are provided through community partnerships with local school districts and community agencies. The Science Festival gives girls the chance to discover science, math, and technology through hands-on activities and meeting women working in industries from all over the science world. Girls Only
Grade Levels
STEM
Afterschool
Summer Camps
Staff PD
STEM Conf
Public STEM
Virtual STEM
Other
Mary Golden mwrgolden@yahoo.com
303-931-2280 http://coolgirls-scienceart.org/contact/ Website: http://coolgirls-scienceart.org/ Mission: To engage young girls in Science, Technology, Engineering, Arts and Mathematics (STEAM) so they have the desire, confidence and skills to sustain their contributions to the community throughout their personal, academic and professional journeys.
!
1-5
!
!
!
Classroom programs that include boys
Description: Cool Girls Science and Art Club helps elementary school children develop their passion for the exciting world of science, technology, engineering, the arts, and mathematics (STEAM). Girlsonly groups meet after school and go on field trips (families welcome!). The girls choose areas of focus and evaluate the program. Each week volunteer science and art professionals mentor the girls and joyfully experiment together.
Appendix A: Selected Afterschool Programs for Comparison to Girlstart
! Other Noteworthy STEM Programs â&#x20AC;&#x201C; (did not meet all our selection criteria, no longer operating, single-focused programs, or programs that include boys):
All Girls Expedition http://www.projectexploration.org/age/ Project Exploration's All Girls Expedition is a summer program that provides an intensive 2-week classroom and fieldwork experience. The expedition begins in Chicago with intensive classroom sessions. Then the team spends one week working in the field alongside scientists. Building Girls http://www.tradeswomen.net/?page_id=12 Oregon Tradeswomen, Inc.â&#x20AC;&#x2122;s Building Girls program helps fill a critical gap for young women by offering ways to explore the variety of exciting, high-paying careers available in the trades. Through supervised hands-on construction experience, positive skills coaching, and working directly with female role models, Building Girls fulfills an unmet community need. Young women who participate in the Building Girls Work Crew are paid for their work and learn early the value of paid on-the-job training. The Building Girls Work Crew is an all female environment where participants learn how to use power tools, frame walls, install metal roofing, learn proper safety etiquette and experience construction work in a positive and supportive environment.
Communication, Science, Technology, Engineering, and Math C-STEM http://www.cstem.org C-STEM started in Houston, TX, operates both in the formal classroom and out-of-school time setting and spans K-12. CSTEM seeks to eliminate barriers to the STEM disciplines as well as communication for under-represented population. The school year culminates with a CSTEM challenge that includes a robotics competition and presentation of work done through the year. CSTM focuses on professional development for teachers who are leading the robotics teams, most often in afterschool time. Each team consists of participants from one high school, middle school and elementary school. Each team completes a section of a larger end product. The different components of the competition include robotics, creative writing, a green challenge, sculpture and GIS mapping.
Girls Math, Science & Engineering http://web.mac.com/sullivlu/iWeb/Site/Welcome.html Part of the Clever Kids Intel Computer Clubhouse Clever Program for youth in Tacoma, WA. No longer funded.
Appendix A: Selected Afterschool Programs for Comparison to Girlstart
! Girls Excelling In Math and Science of Prince George’s County (GEMS-PGC) rarosca@gmail.com An after-school enrichment program for girls attending public middle schools in Prince George’s County. University of Maryland (UMD) College Park Scholars and other undergraduates majoring in math, science and engineering volunteer as GEMS mentors. Activities encourage inquiry-based learning and the practice of science skills. GEMS activities include field trips to science-related research sites and guest presenters from the University and the community. Oceanography Camp for Girls http://www.marine.usf.edu/girlscamp/ A 3-week summer camp for girls entering high school to “learn science while having fun experiencing the ocean world.” Campers get to experience a day at sea aboard a research vessel, coastal marine field trips, “real-world” research projects, career interviews, and visits to local marine attractions. Mentors are graduate students, research scientists, teen counselors from University of South Florida College of Marine Science. TechREACH http://www.techreachclubs.org/index.html TechREACH after school clubs and summer programs help schools involve underrepresented minority and low-income middle school students in technology. TechREACH equips teachers with the proven curriculum and training to generate excitement for science, technology, engineering, and mathematics (STEM) while building students’ confidence. TechREACH is a project of the EdLab Group (formerly the Puget Sound Center for Teaching, Learning and Technology. Girl Game Company http://programservices.etr.org/index.cfm?fuseaction=projects.summary&ProjectID=108 The aim of Girl Game Company (GGC) has been to increase middle school girls’ interest, ability, and motivation to pursue education and careers in IT. The focus is on Latina girls in middle school in the Pájaro Valley Unified School District. GGC met after school and during the summer, over 16 months, for a total of 260 program hours. The program has focused on teaching girls to design and program their own computer games and to produce a 3D animation to introduce one of their games. Another critical focus has been career identity exploration, including social networking with female professional role models in IT fields and field trips to colleges and technology companies.
Appendix A: Selected Afterschool Programs for Comparison to Girlstart
! The Girl Game Company has since evolved into a co-ed, multimedia-focused, information technology (IT) career pathways program serving primarily Latino youth from 5th grade to high school in Watsonville, CA. This expanded program is managed through ETR Associates’ Center for Youth Success and is now called Watsonville Technología-Educación-Comunidad (TEC). Watsonville TEC brings together partners from public education and other STEM-focused afterschool programs, such as Math Engineering Science Achievement (MESA) from the University of California, to create an educational and career pathway that supports students through critical transitions in their secondary school career.
Appendix B: Interview Protocol
1.
First, I would like to confirm with you what I found on your website regarding:
Participants (gender; underrepresented populations)
Grade STEM activities Levels (all four or not)
Afterschool
STEM Education
services)
(conferences, PD, public events,!)
Summer Camp (hours, location of
Other
2. In a few sentences, how do you develop and select your curriculum? Prompts: Established curriculum or created it Modifications for diverse learners and SPED Connections to academics in enrichment activities
3. How do you ensure that the services you provide are of high quality? Prompts: Staff backgrounds Staff professional development Connections to local schools 4. How do you keep/get students engaged in your program? Prompts (To what extent do you have?): Hands on, interactive activities Activities tailored to participantsâ&#x20AC;&#x2122; needs/interests Time for homework; offer homework help Creativity and self-expression Real world activities Outside STEM resources and/or students participate in STEM activities in the field Other ways to engage students
5. What impact has your program had and how do you know it? Prompts: Data on student participants and outcomes Interest in STEM Increased # of STEM classes Data on family involvement Data on community involvement 6. Is there anything else you would like to share about your program?
7. May I contact you if I have follow up questions?
SEDL Evaluation for Girlstart Appendix C: Program Comparison Matrix of Quality Practices for STEM Afterschool Afterschool Program Who Attends (name, location, contact, URL)
Reg. program evaluation
More community involvement
N
Y Y Y Y Y N
Y
Y
Y N Y Y Y N
N
N
Y Y Y Y Y N
Y
N
S S Y Y Y N
S
Y Y Y Y Y Y S Y Y N Y Y Y Y
N
N N Y Y N N
N
Y Y Y N N Y N Y Y N Y Y N n/a n/a n/a n/a N
E.g., Resource guide about summer science programs E.g., Annual Science festival 3 Intent of activities = nonacademic (e.g., activity intent might consist of learning how to work collaboratively but content = academic) 4 School day teachers working as instructors of afterschool program 5 School day teachers assisting with curriculum development and/or planning 6 Other STEM activities include creative projects – language arts integrated with STEM, video game development 7 6 additional programs in community sites 2
More family involvement
6
outcomes, family/community outcomes)
Taking more STEM classes
1
Student Impact of Program (data Engagement collection and use, student More interest in STEM
Y N Y No Af 0 0 5 0 Y Y Y Y 1 90 10- 5 5 N Y N N Y Y data A 13; 25 Y N Y Y Af 6 27 4 3 Y Y Y Y 1 60- 8- 67 40 Y Y N N N Y A 90; 10 120 Cool Girls Art & Science Y N N Y W 0 3 0 0 Y Y Y Y 1 90- 8- 2 3 Y Y N N N Y Club Boulder, CO 120 16
Maintain High Quality Services
Data collected and used
H/ 0 22 2 0 Y Y Y Y 1 60- 25 21 24 N Y N Y Y Y L 75 H/ 0 1 0 0 Y Y Y Y 1 60- 10 1 1 N Y Y Y N Y L 90
Program Curriculum
Outside STEM resources Real world activities Homework time/homework help Student needs/interest-based Hands-on, interactive On school committees Reg. contact w/ school day staff Reg. mtgs. with staff Reg. p.d. for all staff University involvement Other staff* Youth workers Schools day teachers assist with curriculum 5 School day teachers as instructors 4 Mission/goals on website Academic & enrichment 3 Modifications as needed Standards-based Researched-based Purchased curriculum/obtained external sources Other
Project ExplorationSisters 4 Science Chicago, IL Science Club for Girls Cambridge, MS
Other Services Provided
Events 2 YCommunity Education 1 YSummer Camp Ntl. prgm.. Local prgm. Serving a particular community Number of programs currently operating Number of schools Number of weeks in program
Girls Inc. of San Antonio, Y N Y N San Antonio, TX
Amount of minutes
Y N Y N
STEM Afterschool Program Activities Structure
Days per week Math Engineering Technology Science Number of programs for 9–12 Number of programs for 6–8 Number of programs for 1–5 Number of programs for PK–K Predominant ethnicity SPED
Low SES Male Female
Girlstart Austin, TX
Grade Levels
Y Y Y Y Y Y N Y Y N Y Y Y Y
No Y Y Y data Y Y Y Y Y Y N Y Y N Y Y Y No No Y Y Y data data (scie (scie nce nce fair fair) N Y Y Y Y Y N Y Y N Y Y Y Y No No No Y data data data No No No Y data data data