Seesaw ESSA Level IV Report by seesawlearning

Seesaw Logic Model Study Type: ESSA Evidence Level IV

Prepared for: Seesaw Prepared by LearnPlatform: Meetal Shah, Ph.D., Senior Researcher Alexandra Lee, Researcher

April 17, 2023

EXECUTIVE SUMMARY Seesaw engaged LearnPlatform by Instructure, a third-party edtech research company, to develop a logic model for Seesaw. LearnPlatform designed the logic model to satisfy Level IV requirements (Demonstrates a Rationale) according to the Every Student Succeeds Act (ESSA).1 Logic Model A logic model provides a program roadmap, detailing program inputs, participants reached, program activities, outputs, and outcomes. LearnPlatform collaborated with Seesaw to develop and revise the logic model. Study Design for Seesaw Evaluation Informed by the logic model, the next phase will focus on planning for an ESSA Level III study to examine the extent to which Seesaw impacts student learning outcomes. Conclusions This study provides results to satisfy ESSA evidence requirements for Level IV (Demonstrates a Rationale). Study design and planning are currently underway and scheduled for Spring 2023.

Level IV indicates that an intervention should include a “well-specified logic model that is informed by research or an evaluation that suggests how the intervention is likely to improve relevant outcomes; and an effort to study the effects of the intervention, that will happen as part of the intervention or is underway elsewhere…” (p. 9, U.S. Department of Education, 2016). LearnPlatform by Instructure© 2023 Prepared for Seesaw, April 2023

TABLE OF CONTENTS Introduction

Logic Model

Study Design for Seesaw Evaluation

Conclusions

References

Introduction Seesaw engaged LearnPlatform by Instructure, a third-party edtech research company, to develop a logic model for their learning solution. LearnPlatform designed the logic model to satisfy Level IV requirements (Demonstrates a Rationale) according to the Every Student Succeeds Act (ESSA). The Seesaw team recognizes that existing learning tools for PK-6 educators lack the elements that foster student engagement, creative expression, and agency, and fail to meaningfully involve learners' families. They aim to solve these problems by: ensuring all elementary students have access to high-quality instruction with dynamic and developmentally-appropriate, multimodal learning tools; empowering educators with resources to engage in data-driven, personalized instruction; and facilitating a network of support to promote the holistic well-being of children. The study had the following objectives: 1. Define the Seesaw logic model and foundational research base. 2. Draft an ESSA Level III study design. Previous Research. The design of this logic model was guided by previous research examining the research on student engagement via multi-modal learning tools, formative assessment, and feedback. Students engage deeply with learning activities when they believe that they will derive value from and be successful at them (Eccles, J. S., & Wigfield, 2020). Often, students turn away from tasks because success in those tasks is measured through the use of a narrow set of symbolic representations (Jones et al., 2020; Olivier, 2021). This is especially true among younger learners who are still learning how to read and write and students with learning disabilities (Lewis, 2014; Maas & Hughes, 2020). With the proliferation of new technologies, it is now possible to center student voice using multi-modal features (e.g., a video recording of manipulatives that a student used to respond to a equipartitioning (division) problem or an at-home science experiment). Here, the student is provided the opportunity to share their own understanding without having to rely solely on and be held back by symbols. Undoubtedly, students must eventually graduate to symbols and disciplinary lexicon, however, this must be done via careful teacher orchestration that leads them to construct their own shared meaning of a concept and not simply regurgitate a memorized rule or definition (Fyfe et al, 2014; Nathan, 2012). Seesaw recognizes the importance of multi-modal learning tools in the construction of knowledge and as a support for students’ self-efficacy and thus has embedded these in high quality lessons that teachers can readily employ into their daily practice. Seesaw also provides teachers with embedded formative assessments. An important aspect of these assessments for learning are their potential to strengthen student’s metacognition, self awareness, self monitoring, and agency (Heritage et al., 2009). If formative assessment practices are applied expertly, students can learn the skills of self- and peer-assessment such that they collaborate with their teachers in developing a shared understanding of their current learning status and what they need to do to move forward in their learning (Sadler, 1989). Robust formative practices are underpinned by a set of high quality tasks that elicit students’ conceptual LearnPlatform by Instructure© 2023 Prepared for Seesaw, April 2023

understanding (Lotan, 2003) and can identify the gap between what they know and need to know (Heritage, 2007). “If the gap is perceived as too large by a student, the goal may be unattainable, resulting in a sense of failure and discouragement on the part of the student. Similarly, if the gap is perceived as too ‘small,’ closing it might not be worth any individual effort" (Sadler, 1989, p. 130). To close the formative assessment loop, teachers need to provide domain-specific, actionable feedback (e.g., not only responding with a “well done” or “good job”) on hearing or viewing students’ work (Hattie & Timperley, 2007; Kluger & DeNisi, 1996). “Effective feedback provides clear, descriptive, criterion-based information that indicates to the students where they are in their understanding of a concept, how their understanding differs from the desired learning goal, and how they can move forward" (Heritage, 2007, p. 142). This type of specific feedback is also important for promoting a growth mindset because it emphasizes incremental steps to learning (Dickhäuser et al., 2017; Haimovitz & Dweck, 2017). Seesaw has developed mechanisms by which teachers, peers, and family members can easily contribute feedback to students on their activities. The feedback features are designed to elevate students’ ideas by showing them that their contributions to classroom discourse are valued regardless of whether they are “correct” or “incorrect”. Rather than simply receiving a score that can undermine confidence and a growth mindset, such practices provide access to the questions and related thinking that builds student agency and metacognitive processes. Students reflect on their own incremental learning, monitoring what they know and determining when they need more information (i.e., metacognitive monitoring; Schunk, 2008). Thus, students develop and use self-regulated learning strategies and are able to adapt their learning tactics to meet their own learning needs. Formative assessment practices and specific feedback, contribute strongly to building student agency which is a foundation for personalized learning and differentiation. Valid and reliable systems of assessment which assess student mastery of standards, support reflection and continuous improvement, and are also recognized by students as meaningful and valuable to their learning process are often referred to as authentic assessments and are common features of equity-focused competency-based education systems (Sturgis & Casey, 2018). Finally, Seesaw’s portfolios when paired with embedded communication tools for families foster effective family-school partnerships centered on student learning and development (Mapp & Kuttner, 2013). The portfolios are used to collect both formative and summative assessments but also allow for students to showcase their mastery of learning beyond standardized assessments. Teachers can also document student progress and communicate this information with parents on an ongoing basis. Research has shown that engaging families through regular exchanges that are focused on specific academic outcomes helps foster relational trust among teachers, parents, and school leaders and improves academic outcomes for students (Bryk et al., 2010; Henderson & Mapp, 2002, Jeynes, 2012, 2017; and Darling-Hammond & Cook-Harvey, 2018).

Logic Model A logic model is a program or product roadmap. It identifies how a program aims to impact learners, translating inputs into measurable activities that lead to expected results. A logic model has five core components: inputs, participants, activities, outputs, and outcomes (see Table 1).

Table 1. Logic model core components Component

Description

More information

Inputs

What the provider invests

What resources are invested and/or required for the learning solution to function effectively in real schools?

Participants

Who the provider reaches

Who receives the learning solution or intervention? Who are the key users?

Activities

What participants do

What do participants do with the resources identified in Inputs? What are the core/essential components of the learning solution? What is being delivered to help students/teachers achieve the program outcomes identified?

Outputs

Products of activities

What are numeric indicators of activities? (e.g., key performance indicators; allows for examining program implementation)

Outcomes

Short-term, intermediate, long-term

Short-term outcomes are changes in awareness, knowledge, skills, attitudes, and aspirations. Intermediate outcomes are changes in behaviors or actions. Long-term outcomes are ultimate impacts or changes in social, economic, civil or environmental conditions.

LearnPlatform reviewed Seesaw resources, artifacts, and program materials to develop a draft logic model. Seesaw reviewed the draft and provided revisions during virtual meetings. The final logic model depicted below (Figure 1) reflects these conversations and revisions.

Seesaw

Logic Model Problem Statement : Existing learning tools for PK-6 educators lack the elements that foster student engagement, creative expression, and agency, and fail to meaningfully involve learners' families. Seesaw aims to solve these problems by: ensuring all elementary students have access to high-quality instruction with dynamic and developmentally-appropriate, multimodal learning tools; empowering educators with resources to engage in data-driven, personalized instruction; and facilitating a network of support to promote the holistic well-being of children.

Inputs

Activities

Outputs

Outcomes

What does 'Product' provide?:

How are participants using 'Product'?

What are the measurable results of implementing 'Product'?

What are the expected changes or impacts of using 'Product'?

STUDENTS - Complete teacher-assigned activities/lessons using multimodal learning tools - Receive feedback from peers, educators, and family members on work - Document offline learning activities and add layers of explanation and reflection via photos, videos, voice recording, etc. - Showcase mastery of learning through their portfolio of work - Complete teacher-assigned formative assessments

STUDENTS Number and nature of: - activities/lessons completed and multimodal tools used - feedback received from and provided to peers - feedback received from educators and family members - posts of offline activities Number of: - portfolio views - formative assessments completed - sessions/logins (total)

SHORT-TERM Students: - report deeper engagement with learning; ability to contribute to and develop a shared understanding of content, and greater agency, - have greater opportunities to connect with and learn from peers, educators, and family - have increased self-awareness, including strengths and areas for growth - have a growth mindset in learning - have increased proficiency on assessments Educators - use of developmentally-appropriate instructional resources increases - use of formative assessments increases - report greater access to resources for planning, instruction, and interventions - report feeling increased partnerships with families Administrators - have access to data to support decision-making Families - report having greater visibility and understanding of classroom happenings

'PRODUCT' PROVIDES - Supplemental curriculum - Lesson creation and delivery platform with multimodal learning tools - Resource library of lessons and activities - Assessment, progress monitoring, and portfolio tools - Integrated communication platform - LMS and SIS integrations - Teacher professional development (PD)

DISTRICT PROVIDES - Internet-enabled devices - Seesaw subscription - Curated ?District and School Library? in Seesaw that organizes lessons and activities aligned to the district?s specific scope & sequence/pacing guides (optional)

Participants Who uses 'Product'? - Pre-K - 6 Students - Pre-K - 6 Educators - Families - District and School Administrators

EDUCATORS - Create and/or assign activities/lessons - Provide personalized feedback to students in text or audio recordings - Differentiate instruction e.g. targeted interventions, scaffolds, and small group activities - Enable students to document and extend offline learning activities - Review student engagement and performance portfolios, standards-aligned skills, and activity-based reports - Document mastery of learning through students' portfolio of work - Communicate with families about classroom happenings and student progress - Assign formative assessments ADMINISTRATORS - Support educators with targeted PD and advocate for Seesaw use - Virtually visit classrooms to gain insight into instruction and student progress - Communicate with teachers, students, and families

FAMILIES - Review different levels of communications from school to facilitate supporting learning at home - Communicate with school leaders and teachers about student needs, with built-in translation available in 100+ languages - See their child?s learning in the Seesaw Journal and post encouraging comments - Review and celebrate mastery of learning through their child's portfolio of work

EDUCATORS Number and nature of: - activities/lessons created and/or assigned - feedback provided to students - differentiated instruction via individual or group lessons/activities assigned - messages sent to families - posts on classroom happenings Number of: - of portfolio and report views - of PDF reports sent to families - of formative assessments assigned

ADMINISTRATORS - Number of school communications opened/read - Number and nature of feedback given to their child

FAMILIES - Number and nature of PD opportunities provided for educators - Number of class-, school-, or district-level report views - Number and nature of messages sent to families - Number of portfolio views and engagement with posts

INTERMEDIATE Students' - value for learning activities in the classroom increases - confidence and self-efficacy as learners improves - content-based skills improve - scores on standardized assessments increase Educators' - ability to deliver data-driven and effective instruction improves - ability to differentiate instruction is more effective Administrators - support the ongoing use of Seesaw through PD, curriculum, and school-wide communication - proactively leverage engagement and assessment data to address inequities Families - report having greater confidence and resources to support their students' learning progress

LONG-TERM Students - report greater belonging to their community - are better prepared to tackle middle school coursework Educators - well-being and retention improves - have sustained engagement with the Seesaw platform and other supporting edtech - are better positioned to close gaps in student learning outcome Administrators - enact a high-quality, equitable curriculum across classes

Seesaw Logic Model Components. Seesaw invests several resources into their program, including supplemental curriculum; lesson creation and delivery platform with multimodal learning tools; resource library of lessons and activities; assessment, progress monitoring, and portfolio tools; integrated communication platform; learning management systems (LMS) and student information systems (SIS) integrations; and teacher professional development. School/Districts are expected to provide internet-enabled devices and the Seesaw subscription service. They can also optionally provide a curated “District and School Library” in Seesaw that organizes lessons and activities aligned to the district’s specific scope and sequence documents or pacing guides. Ultimately, the Seesaw program aims to reach Pre-K - 65 students, Pre-K - 6 educators, families, and district and school administrators. Using these program resources, the participants can engage with the Seesaw platform in the following activities: ● Pre-K - 6 students: ○ complete teacher-assigned activities/lessons using multimodal learning tools; ○ receive feedback from peers, educators, and family members on work; ○ document offline learning activities and add layers of explanation and reflection via photos, videos, voice recording, etc.; ○ showcase mastery of learning through their portfolio of work ; and ○ complete teacher-assigned formative assessments. ● Pre-K - 6 educators: ○ create and/or assign activities/lessons; ○ provide personalized feedback to students in text or audio recordings ; ○ differentiate instruction e.g. targeted interventions, scaffolds, and small group activities; ○ enable students to document and extend offline learning activities; ○ review student engagement and performance portfolios, standards-aligned skills, and activity-based reports; ○ document mastery of learning through students' portfolio of work; ○ communicate with families about classroom happenings and student progress; and ○ assign formative assessments. ● District and School Administrators: ○ support educators with targeted PD and advocate for Seesaw use; ○ virtually visit classrooms to gain insight into instruction and student progress; and ○ communicate with teachers, students, and families ● Families: ○ review different levels of communications from school to facilitate supporting learning at home; ○ communicate with school leaders and teachers about student needs, with built-in translation available in 100+ languages; and ○ see their child’s learning in the Seesaw Journal and post encouraging comments.

Seesaw can examine the extent to which core activities were delivered and participants were reached by examining the following quantifiable outputs: ● Pre-K - 6 students: ○ Number and nature of: ■ activities/lessons completed and multimodal tools used; ■ feedback received from and provided to peers; ■ feedback received from educators and family members; and ■ posts of offline activities ○ Number of: ■ portfolio views; ■ formative assessments completed; and ■ sessions/logins (total). ● Pre-K - 6 educators: ○ Number and nature of: ■ activities/lessons created and/or assigned; ■ feedback provided to students; ■ differentiated instruction via individual or group lessons/activities assigned; and ■ messages sent to families. ○ Number of: ■ of portfolio and report views; ■ of PDF reports sent to families; and ■ of formative assessments assigned. ● District and School Administrators: ○ Number of school communications opened/read ○ Number and nature of feedback given to their child ● Families: ○ Number and nature of PD opportunities provided for educators ○ Number of class-, school-, or district-level report views ○ Number and nature of messages sent to families If implementation is successful, based on a review of program outputs, Seesaw can expect the following short-term outcomes. Students will report deeper engagement with learning; ability to contribute to and develop a shared understanding of content, and greater agency. They will also have: greater opportunities to connect with and learn from peers, educators, and family; increased self-awareness, including strengths and areas for growth; a growth mindset in learning; and increased proficiency on assessments. Educators' use of developmentally-appropriate instructional resources and formative assessments will increase in the short term. They will also report: greater access to resources for planning, instruction, and interventions and feeling increased partnerships with families. Administrators will have access to data to support decision-making and families will report having greater visibility and understanding of classroom happenings. LearnPlatform by Instructure© 2023 Prepared for Seesaw, April 2023

In the intermediate term, students' value for learning activities in the classroom will increase. Similarly, their confidence and self-efficacy as learners will improve. Subsequently, their content-based skills and scores on standardized assessments will also improve. Educators' ability to deliver data-driven, effective, and differentiated instruction will improve. Administrators will be better able to support the ongoing use of Seesaw through PD, curriculum, and school-wide communication and proactively leverage engagement and assessment data to address inequities. While families will report having greater confidence and resources to support their students' learning progress. Ultimately, in the long term, Seesaw expects students to report greater belonging to their community and be better prepared to tackle middle school coursework. Educators’ well-being and retention improves and they will have sustained engagement with the Seesaw platform and other supporting edtech. Thus, educators will be better positioned to close gaps in student learning outcomes. Administrators will enact a high-quality, equitable curriculum across classes.

Study Design for Seesaw Evaluation To continue building evidence of effectiveness and to examine the proposed relationships in the logic model, Seesaw has plans to conduct an evaluation to determine the extent to which its program produces the desired outcomes. Specifically, Seesaw has plans to begin an ESSA Level III study to answer the following research questions: Implementation Questions 1. To what extent did students use Seesaw during the 2021–22 (or 2022-23) school year? ○ What is the average number of activities or lessons that students completed on the Seesaw platform? ○ What multimodal tools did students use when engaging with activities or lessons? ○ On average, how much feedback did students receive from their educators on activities or lessons? Outcome Questions After controlling for prior achievement and other student demographic characteristics (e.g. ELL status, FRL status, gender, and race/ethnicity), 2. did performance on Seesaw formative [subject area e.g., math/ELA/science] assessments vary based on students’ usage of Seesaw during the 2021–22 (or 2022-23) school year? 3. did performance on standardized [subject area e.g., math/ELA/science] assessments vary based on students’ usage of Seesaw during the 2021–22 (or 2022-23) school year? Seesaw plans to begin this study in spring 2023.

Conclusions This study satisfies ESSA evidence requirements for Level IV (Demonstrates a Rationale). Specifically, this study met the following criteria for Level IV: Detailed logic model informed by previous, high-quality research Study planning and design is currently underway for an ESSA Level I, II or III study

✅ ✅

References Bryk, A. S., Sebring, P. B., Allensworth, E., Easton, J. Q., & Luppescu, S. (2010). Organizing Schools for Improvement: Lessons from Chicago. Chicago, IL: University of Chicago Press. Darling-Hammond, L., & Cook-Harvey, C. M. (2018). Educating the whole child: Improving school climate to support student success. Palo Alto, CA: Learning Policy Institute. Dickhäuser, O., Janke, S., Praetorius, A. K., & Dresel, M. (2017). The effects of teachers' reference norm orientations on students' implicit theories and academic self-concepts. Zeitschrift für Pädagogische Psychologie. Eccles, J. S., & Wigfield, A. (2020). From expectancy-value theory to situated expectancy-value theory: A developmental, social cognitive, and sociocultural perspective on motivation. Contemporary Educational Psychology, 61. Fyfe, E. R., McNeil, N. M., Son, J. Y., & Goldstone, R. L. (2014). Concreteness fading in mathematics and science instruction: A systematic review. Educational Psychology Review, 26, 9-25. Haimovitz, K., & Dweck, C. S. (2017). The origins of children's growth and fixed mindsets: New research and a new proposal. Child Development, 88(6), 1849-1859. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. Henderson, A. T., & Mapp, K. L. (2002). A new wave of evidence: The impact of school, family, and community connections on student achievement. Austin, TX: National Center for Family & Community Connections with Schools. Heritage, M. (2007). Formative assessment: What do teachers need to know and do?. Phi Delta Kappan, 89(2), 140-145. Heritage, M., Kim, J., Vendlinski, T., & Herman, J. (2009). From evidence to action: A seamless process in formative assessment?. Educational Measurement: Issues and Practice, 28(3), 24-31. Jeynes, W. H. (2012). A meta-analysis of the efficacy of different types of parental involvement programs for urban students. Urban Education, 47(4), 706–742. Jeynes, W. H. (2017). A meta-analysis: The relationship between parental involvement and Latino student outcomes. Education and Urban Society, 49(1), 4–28. 54. Jones, P., Turney, A., Georgiou, H., & Nielsen, W. (2020). Assessing multimodal literacies in science: semiotic and practical insights from pre-service teacher education. Language and Education, 34(2), 153-172.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254. Lewis, K. E. (2014). Difference not deficit: Reconceptualizing mathematical learning disabilities. Journal for Research in Mathematics Education, 45(3), 351-396. Lotan, R. A. (2003). Group-worthy tasks. Educational Leadership, 60(6), 72-75. Maas, M. J., & Hughes, J. M. (2020). Virtual, augmented and mixed reality in K–12 education: A review of the literature. Technology, Pedagogy and Education, 29(2), 231-249. Mapp, K. L., & Kuttner, P. J. (2013). Partners in education: A dual capacity-building framework for family–school partnerships. Austin, TX: SEDL & U.S. Department of Education. Retrieved from http://www2.ed.gov/documents/family-community/partners-education.pdf Nathan, M. J. (2012). Rethinking formalisms in formal education. Educational Psychologist, 47(2), 125–148. Olivier, J. (2021). Self-directed multimodal assessment: Towards assessing in a more equitable and differentiated way. Learning through assessment: An approach towards Self-Directed Learning. NWU Self-Directed Learning Series, 7, 51-69. Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. Schunk, D. H. (2008). Metacognition, self-regulation, and self-regulated learning: Research recommendations. Educational Psychology Review, 20, 463-467. Sturgis, C., & Casey, K. (2018). Designing for Equity: Leveraging Competency-Based Education to Ensure All Students Succeed. CompetencyWorks Final Paper. iNACOL.