Annex C Endline evaluation matrix, part I

Table 25: Endline evaluation matrix part I (partial, related to quantitative evidence)

Prog. Comp. Results chain level Descriptive link in theory of change

All impact Better learning outcomes, especially for girls

Contributions from all component result chains reinforced by interlinkages. Strong governance model (community monitoring to school leadership to ward co-ordination to district management to regional strategic leadership), acts as a foundation for impact and sustainability.

Endline evaluation questions (partial, to be answered using quantitative evidence)1

Did standard 3 pupil learning in Kiswahili and maths improve? Why? To what extent can this be attributed to EQUIP-T? Did learning gaps narrow for marginalised groups (girls, nonKiswahili speakers)? Why?

All outcome

Better quality education, especially for girls (positive, inclusive and girl-friendly learning environment in schools); improved retention rates for girls and boys2

No specific additional questions related to ‘better quality education, especially for girls’ beyond those captured under the components below; all components contribute to this outcome, and the evidence will be synthesised to get a holistic picture.

All intermediate outcome to outcomes and impact (assumption)

Teacher performance

DEMAND-SIDE SUPPORTING CONDITIONS RIGHT: regular pupil attendance; school-ready children/attendance at pre-school; adequate support at home for school work; positive parental attitudes to education for girls/CWD/OVC; other social-economic conditions do not pose overwhelming barriers; appropriate class sizes

C1A: Tchs input INSET for early grade (EG) teachers; teaching and learning materials (TLM) provided; positive/safe learning campaign materials provided; info. on structures for teachers’ communities of learning (COL) provided

C1A: Tchs output EG teachers' capacity, confidence increased; TLM & positive learning campaign materials available in classrooms; COL structures operating

C1A: Tchs input to output (assumption)

Teachers readiness to learn; relevance/accessibility of INSET & TLM materials; teacher attendance at INSET; quality of INSET maintained from district/ward level to school level; gender-based attitude/ customs are open to change

Did assumptions on pupil attendance, pre-school attendance, support at home, hold? Have there been changes over time? Has pupils’ household poverty status changed over time?

Did EG teachers receive district/ward INSET as intended? Did school-based INSET happen as intended? Did school receive materials as intended?

Did EG teachers' capacity (curriculum knowledge, general/inclusive pedagogical skills) increase? Are teachers more confident? Are materials in classrooms? Are teacher COL structures operating?

Did assumptions on INSET attendance hold? What challenges, if any, do teachers face in attending and learning from INSET?

Prog. Comp. Results chain level Descriptive link in theory of change

C1A: Tchs intermediate outcome Classroom teaching improved; materials being used effectively; teachers using positive behaviour management; instructional hours appropriate

C1A: Tchs output to intermediate outcome (assumption)

School leadership and management

SUPPLY-SIDE SUPPORTING CONDITIONS RIGHT including high teacher school & classroom attendance; punctuality; low teacher turnover; limited movement of EG teachers to upper stds; high morale/motivation and job satisfaction

C2A/B: SLM input HT/WEO training on SLM, school development planning (SDP), school performance management meetings (SPMMs), school information system (SIS); tablets delivered; info. on structures for HTs’ COL provided; WEOs with transport and grants (from C3A)

C2A/B: SLM output HT capacity (knowledge/skills) increased; annual SDPs that meet quality criteria available; SPMMs happen weekly & teachers share problems/solutions; HTs meet as COL & share problems/solutions; SIS is functional; produces useful reports to support SLM; effective monitoring/support from WEOs

Endline evaluation questions (partial, to be answered using quantitative evidence)1

Has classroom teaching improved (more active, inclusive, genderresponsive & use of TLM)? Are teachers using more positive behaviour management? Have instructional hours increased?

Did assumptions on attendance, punctuality, turnover, morale hold? If not, why? Have there been changes over time? Are supporting conditions for teachers (e.g. overall workload, outstanding payments due, housing etc.) improving?

Did HTs receive training as intended? Did schools receive tablets?

Has HT SLM capacity increased in core areas (planning, tch & resource m/ment, community engagement, communication/ reporting)? Are ‘quality’ SDPs available for the current year (evidence-based, costed, incl. teaching and learning & inclusive education activities)? Are SPMMs happening? How frequently? What is discussed? Are HTs meeting as COLs? Is the SIS up-to-date with pupil/teacher records? Is daily attendance being recorded? Can HTs produce useful analysis/reports from the SIS? Have WEO visits increased? Are there any changes in what WEOs are doing during visits?

C2A/B: SLM input to output (assumption) HT/WEC readiness to learn/ability to use technology; relevance/accessibility of INSET materials; SC/communities have time to be actively engaged in SDP; HT/WEO attendance at INSET; HT/teacher attendance at SPMMs; HT attendance at COL meetings; SIS software is well designed for purpose; low turnover of WEOs

C2A/B: SLM intermediate outcome HT leads school more effectively by applying new skills and knowledge (positive changes in attitude and confidence); SDPs are implemented; HT takes action based on SIS; greater school transparency

Did assumptions on attendance hold? If not, why?

Has SLM improved? In which areas? What actions has the HT taken to improve the school? Have some activities from SDPs been implemented? Does the HT use the SIS for SLM? Does the HT make public key information on pupil performance, teacher/pupil attendance & resource use?

Prog. Comp. Results chain level Descriptive link in theory of change

C2A/B: SLM output to intermediate outcome (assumption)

High level of HT school attendance and punctuality; low turnover of HTs; HT morale/motivation and job satisfaction is high; monthly capitation grants received in full; little shortfall in planned resources receipts

Community participation and demand for accountability

C4A Com input School committee(SC)/Parent-teacher partnership (PTP) training; PTP grant#1; business plan training; IGA grant; community needs assessment (CENA) conducted; notice boards distributed

C4A Com output SC capacity (knowledge/skills) improved; SCs active; PTPs set up & active; PTP grant spent; capacity for school-community IGA built; IGA grant spent; CENA action plan in place & linked to SDP; school noticeboards publically accessible with relevant information

C4A Com input to output (assumption)

Training materials/approach are accessible/relevant; attendance at training/CENA process is high; SC/PTP members have time & motivation to change/engage; PTPs able to self-organise; no conflict between SC & PTPs; HTs open to increased transparency/scrutiny;

Endline evaluation questions (partial, to be answered using quantitative evidence)1

Did assumptions on attendance, punctuality, turnover, morale/motivation hold? If not, why? Have there been changes over time? Are supporting conditions for HTs (e.g. overall workload, outstanding payments due, housing etc.) improving? Are capitation grants received monthly in the expected amounts? What other sources of funding does the school have?

Was training received as intended? Did schools receive grants? Did schools receive noticeboards?

Is the role of the SC and PTP clear and distinct? Do SCs meet regularly? What do SCs discuss? Have PTPs been set up, and are they active? Have the grants received been fully spent, and on what? Are HTs aware of CENA action plans? Do schools have publically accessible noticeboards? What is displayed on them?

See footnote (1)

C4A Com intermediate outcomes

Better engagement between schools & all parents/wider communities (mutual understanding, actions & support for school improvement); improved communication between schools & all parents/wider communities; parents/ communities hold schools to account

Have PTPs taken action to improve education in last school year? Are all parents aware of PTPs? Has an IGA been established, using the grant? Is the HT aware of any activities resulting from the CENA action plan? Do parents read the school noticeboard? Do parents receive regular information on the progress of their children? Do parents receive information in other ways?

C4A Com output to intermediate outcome (assumption)

PTP & community members have time, resources & motivation to support schools; free education policy does not undermine community motivation; parents visit schools regularly; no misuse of funds; no political interference

See footnote (1)

Prog. Comp. Results chain level Descriptive link in theory of change

C4B Com input Tchs/PTP members training on JUU clubs; PTP grant#2 for girls’ education; Shujaaz magazine & radio campaign

C4B Com output JUU clubs set up and active; PTP grant#2 spent; Shujaaz magazine distributed to the target group

C4B Com input to output assumptions

Training materials/approach are accessible/relevant; attendance at training is high; little resistance in the school community to increasing openness around taboo subjects; PTP members have time & motivation to plan grant activities for girls’ education

Endline evaluation questions (partial, to be answered using quantitative evidence)1

Did JUU club training take place? Did schools receive the grant? Did school receive Shujaaz magazines?

Do schools have JUU clubs? Do they meet regularly? Has the grant been spent?

See footnote (1)

C4B Com intermediate outcome

Better support for girls, & other marginalised groups incl. CWD, OVC, to be safe, attend school regularly, and learn; better community engagement & support for education for girls & other marginalised children

What actions have the JUU club taken in the last school year? What was the PTP grant#2 spent on? What are the main issues discussed between the school and all the parents when they meet?

C4B Com output to intermediate outcome assumptions little resistance in the school community to increasing openness around taboo subjects and taking related action; no misuse of funds; no political interference

Source: OPM 2018a. Notes: (1) The blank cells in the column under Component 4 mean that no quantitative evidence will be collected related to the assumptions described. These assumptions have been included for completeness, but require qualitative methods to capture (as was done at the midline). (2) Retention rates are not captured in the quantitative IE survey. The IE team will explore EMIS data (publically available on the government’s opendata website) to see if any trend data in primary school retention rates for the 5 original EQUIP-T regions (and the 17 districts within these regions that the EQUIP-T IE survey estimates represent) is available, and could be presented as additional context for the evaluation findings.