SELCO Digital Education Program by IDEX Insights & Fellow Publications

SELCO Digital Education Program

Social Impact Assessment Guide Evelyn NAMARA JUNE, 2015

A report submitted to SELCO team to help in the social impact evaluation of the program.

Executive Summary This report analyses SELCO’s eSHALA Digital Education Program which is an intervention that gives schools in rural areas a chance to operate multimedia programs using a solar enabled digital setup. The report looks at key in-depth analysis of the program using a case study of two schools visited, and highlights different stakeholder involvement and areas of improvement for various stakeholders. The report also covers recommendations to improve metrics for measuring Impact for the Digital Education Program.

Comprehensive Guide for eShala DEP Social Impact Assessment

Table of contents 1. Introduction

1.1 SELCO

1.2 Digital Education Program

1.3 Problem Statement

2. Social Impact Assessment Program

2.1 Stakeholder Analysis

2.2 Analysis of project context

2.3 Analysis of Project Objectives

2.4 Consultation with Stakeholders

2.4.1 Engaging Teachers

2.4.2 Engaging Students

Data analysis and key recommendations

3.1 On site assessments and recommendations

Creating Impact Assessment Metrics

4.1 An outlook on Outputs, What gets measured?

4.2 Measuring eShala DEP Outcomes

Conclusion

Abbreviations

Appendix: Stakeholder Questionnaires

Comprehensive Guide for eShala DEP Social Impact Assessment

Introduction

1.1 SELCO SELCO Foundation envisions a socially sustainable society. They seek to create avenues for asset building, enhancement in quality of life and wealth creation that will uplift deprived sections of society through sustainable development solutions. SELCO foundation’s mission is to; 1. Systematically identify diverse needs of underserved communities, understand and define the role of sustainability and energy in these communities. 2. Create and support innovative and sustainable solutions that positively impact well-being, education and livelihoods and work towards the alleviation of poverty. 3. Foster ecosystem development in the social sector through holistic thought processes in technology, finance, entrepreneurship and policy.

1.2 eSHALA Digital Education Program Sustainable energy practitioners have long noticed energy access affects rural and urban populations’ livelihood, generation capacity, education attendance, health outcomes and overall quality of life. In line with SELCO’s mission, the Digital Education Program was born with a mission to give rural schools a chance to operate independent of the grid malfunctioning. The Digital Education Program targets schools in rural areas that demonstrate a need for quality education but yet can not afford the cost that comes with incorporating multimedia in their education programs with an end goal of achieving quality education.

1.3 Problem Statement Initial feedback suggests that the goal of the eSHALA Digital Education Program is to improve quality of classroom education at the same time helping students understand and retain concepts leveraging on a different pedagogy that involves use of Multimedia. SELCO partnered with CLT India as the key content providers to train and deliver multimedia content using a mini computer and a projector.

Comprehensive Guide for eShala DEP Social Impact Assessment Â

After one year of partnership with CLT India, an assessment to measure impact and look at areas of improvement has not been made. No impact assessment has been done both to determine if SELCO and CLT India are meeting the intended outcomes of the project. This report analyses the project after one year and makes recommendations to improve metrics for assessing impact.

3Â

Comprehensive Guide for eShala DEP Social Impact Assessment

Social Impact Assessment Process

2.1 Stakeholder Analysis Since 2010, SELCO Foundation has expanded to create a holistic ecosystem to provide reliable energy services, for the benefit of impoverished segments in society. The Foundation works to link the benefits of sustainable energy to poverty eradication by collaborating with NGOs, local financial institutions, education institutions and social enterprises. Education is a big part of SELCO’s mandate and the Digital Education Program provides the opportunity for SELCO to work with other stakeholders to bring about quality education in low income populations using sustainable energy solutions. SELCO foundation funds 50% of the DEP implementation which includes setting up and maintaining the solar equipment that powers the multimedia setup, other stakeholders in the DEP project include; School Administration SELCO primarily targets low income schools for the implementation of its DEP program. They approach the school administrations of these schools and sell them the idea of involving multimedia instruction in their schools. Each participating school has to fund 50% of the project for them to have the installation done in their schools. ●

What schools get as part of the project ➔ Full multimedia installation consisting of a solar setup (panels, batteries), LED Projector and mini computer. ➔ Training both on the technical aspects of the setup as well as training on how solar enables their setup. ➔ Content for Maths and Science pre-loaded on the mini computer. ➔ Tablets to help teachers with their lesson plans and execution of lessons. Expectations from Schools ➔ Ownership and overall safety of the set up ➔ Constant use of the system for subjects that are covered within the project ➔ Feedback to SELCO and CLT India about what improvements need to be done ➔ Monitoring impact in terms of children's learning outcomes and the potential that multimedia brings to a classroom 4

Comprehensive Guide for eShala DEP Social Impact Assessment

CLT India CLT identified a problem in the education sphere within India which is that children face many challenges while engaging in learning; the most intimidating of them all is not having a role to participate in the learning process. ●

On the teachers’ front, as much as the other non-teaching duties come in the way of good preparations, lack of subject expertise and teaching resources and over-crowded classes are few of the reasons that limit the teachers to a delivery mode of lessons, rather than engaging children in partnership for learning. It’s because of this background that CLT is expanding learning opportunities and giving access to quality education for children in under-served communities. Technology has been a fundamental enabling factor in CLT’s implementation process. To counter and respond to shortage of elementary school teachers and resource-starved government schools, CLT has engaged in sourcing teachers with subject expertise and digitally capturing their pedagogical practices. CLT’s Digital educational content can be accessed via mobile devices such as Android mini-PCs, Tablets and mobile phones. CLT Partnership with SELCO CLT India are the content partners for the DEP Project, since setups are done in low income communities with no access to internet, CLT offers its offline content that is accessed through the Android mini-PCs. They provide content for 5th - 10th grades based on NCF guidelines and customized for Karnataka state syllabus and mapped to NCERT syllabus. For DEP project, they provide Maths, English and Science content to schools. What CLT offers as part of DEP project ➔ Android Mini-PCs pre-loaded with Maths & Science content ➔ Lesson plans for teachers ➔ Tablets for teachers to access their lesson plans ➔ Training for teachers on how to use the equipment and how to conduct classes using multimedia ● Parents / Community Having a multimedia setup within low income government schools builds long term relationship within the community. Parents reap the benefits of having well empowered children who have access to technology and whose learning outcomes improve with time.

Some of the benefits of using the multimedia setup is that children constantly take home science experiments that they perform at home and parents get to see the tangible benefits of the program.

Comprehensive Guide for eShala DEP Social Impact Assessment

2.2 Analysis of project context The Digital Education Program is implemented in 120 schools across Bangalore and the outskirts. SELCO identifies the schools according to need and location and provides 50% of the costs associated with the setup. Each school selected is required to cover the other half of 50% to get the digital multimedia setup in their schools. This model allows for ownership and buy in from schools and in the long run leads to better management and care of the equipment. The multimedia equipment installed by SELCO involves the following; ● An LED projector ● A licensed mini computer which is android powered ● A solar setup consisting of panels and a battery pack The schools are responsible for the overall safety of the equipment. The equipment is normally installed in classes that are presumed to be the safest and the school allocates someone in charge of the overall safety. The LED projector and Mini computer

Figure 1: DEP installation in Kopal Vidya Kendra School The mini computer is pre-loaded with content from CLT India who are the content providers / academic partner for the Digital Education Program. The loaded mini computer is then attached to 6

Comprehensive Guide for eShala DEP Social Impact Assessment

the LED projector and is powered by solar. The setup also has speakers attached to the mini computer to provide additional sound. The installation is static which means that the equipment can not be moved around. The projector projects on a projector screen in front of a classroom, there’s supposed to be a blackboard next to the screen to allow teachers to explain concepts seen on the multimedia set up in depth.

Figure 2: The projector screen The Solar setup SELCO as an organisation has a focus on serving the unserved and underserved and one of their core areas is energy access. They install and maintain the solar equipment at each of the partner schools. One biggest aspect after the installation is the education bit. SELCO ensures that the teachers who use the Digital Education equipment are trained both on the technical aspects of the set up but also the fundamentals of solar technology. In each corner where the solar batteries are kept, SELCO has a poster that explains how Solar works and how the setup is powered. The teachers in turn educate the students who come for the multimedia class about how solar works and how the equipment is powered. This solar education creates great impact for SELCO by extending the knowledge and importance of using sustainable energy in the minds of the students. SELCO also leaves behind contacts for emergency situations but on a greater scale they train the teachers to be able to deal with any troubleshooting needs they may encounter with the setup.

Comprehensive Guide for eShala DEP Social Impact Assessment

Figure 3: Poster explaining how solar works The poster above explains how solar works in Hindi, and is a great tool for educating local communities about the importance of using sustainable energy. Setup concerns Due to the fact that the setup is static, safety issues have to be looked at. How do the schools ensure that there’s adequate safety for the equipment? Who takes ownership of safety? Another fundamental aspect that should be noted from the set up is the need to keep the equipment safe from dust. Since the equipment can not be un installed and kept safely, it harbours a lot of dust that comes from the daily cleaning of the classroom. In the long run, if the equipment is constantly exposed to dust, it may stop functioning properly.

Comprehensive Guide for eShala DEP Social Impact Assessment Â

2.3 Analysis of Project Objectives The DEP project ultimately seeks to empower many low income schools with tools and resources to enable them enhance and improve on the skills development of children who access the program. The more schools empowered through the project, the bigger the impact in the community.

Many low income government schools are faced with outdated pedagogical methods of teaching and as a result, students come out of such schools without any critical thinking skills, and they are not able to attain jobs. Programs like DEP should help bridge the gap by enhancing teaching practices to help get students curious and better at their skills due to the experiments carried out in class.

2.4 Consultation with Stakeholders For this report, I consulted with different stakeholders to understand how the implementation of DEP was on ground. SELCO provided enough background about their involvement and I visited two implementing schools of DEP. Case study: Gopal Vidya Kendra This school is located in the village of Soladevana Halli, North 4 Block, Bangalore North District, and has both Kannada and English mediums. The school was established in 2008 and is a private school located in a low income settlement.

9Â

Comprehensive Guide for eShala DEP Social Impact Assessment

SELCO partnered with this school to bring the “Smart Class” to the community and empower students with a better education and a fun way of teaching using multimedia.

Figure 5: One of SELCO’s DEP implementing schools

2.4.1 Engaging teachers I spoke with 2 teachers of Maths and Science. Both of them had high praises of how the smart class has empowered them to be better teachers. Below are some of the questions we discussed in our meeting; - What their overall feelings were about incorporating multimedia in their class - How relevant the content from CLT is compared to the syllabus - How they prepare for their classes - How much training they got before using the equipment - Their views on lesson plans and how easy or difficult it is to prepare these - Do they see any behavioral changes in the students since using smart class? - Is there an improvement in Students’ grades since using multimedia? Majority of the answers from teachers showed that they saw this as a positive inclusion to their classes. They felt empowered using technology to conduct their classes and were certain this way of teaching would bring about better results in students grades. Positive outcomes - Students appeared early for their classes because of the excitement of using an alternative learning method They see improvement in the student's participation in class -

The students are able to grasp concepts much better 10

Comprehensive Guide for eShala DEP Social Impact Assessment

The learning is self paced and the teacher can play the video so many times for them to understand

Availability of multiple examples helped reinforce learning outcomes “Learning by seeing” - helps students retain most of the knowledge They are able to go through their lesson plans and plan for classes using the tablets provided by the project Content is easy to understand and it is engaging for the students

Areas of improvement - More content needs to be added on the tablets - not the entire syllabus is loaded - The teachers felt that more videos need to be added - The sound used to record the videos is so low and sometimes students at the back of the class are not able to hear - They felt that the equipment should be open so that they can add more data or use the setup for other educational activities

2.4.2 Engaging Students The students loved the inclusion of technology and wanted the smart class to be rolled out to all subjects not just Maths and Science. The setup helped them understand concepts better and they were able to play the video again if they didn’t understand the concept. Below are some of the talking points I had with the students; - Do they understand why the setup is important? - How many of them understand solar - and how the equipment is powered - How has technology enhanced their way of learning? - What is the significant difference between using multimedia and their other “normal” classes? - How have they improved in terms of learning? - Are they able to carry out experiments done in class at home? - Do they carry out experiments seen in videos by themselves? A lot the responses to the following were positive with students saying that they enjoy the digital classes more because they are interactive. A lot of them also pointed out that they tried out experiments back home but some faced a challenge where they were not able to get the necessary equipment to complete most experiments, in other words the equipment mentioned in videos is not easily accessible in their communities. My observations above are based on 2 schools I visited and both schools have only been using the setup for 6 months. A lot of improvements need to be made both on the setup point of view but also from the Content provider's point of view. The next chapter digs in depth into the preliminary findings.

Comprehensive Guide for eShala DEP Social Impact Assessment

Data analysis and key recommendations

3.1 On site assessments and recommendations From data collected during the stakeholder engagements, a meeting with a SELCO representative provided great insights into the eShala Digital Education program. The project is relatively new and has been implemented for over a year in 120 schools. Number of trainings conducted; The teachers are provided with one training from SELCO and one from CLT. SELCO is responsible for setting up the equipment in schools and gives an initial demo of using the system to teachers. CLT India who are the content providers then provide a much detailed training on the use the digital content as part of their pedagogy to have the desired impact. Only one training is conducted at the beginning. ●

There’s a need for teachers to have more additional trainings in intervals to refresh their memory. From my conversations with teachers, there’s no clear strategy in place for training teachers who join in later and missed the initial training. Both the school administration, SELCO and CLT India have to work out a training program that addresses training needs for schools. Things to consider; -

Measuring the teachers retention of data through their lesson planning, execution and ability to train new teachers A clear map of who handles training in house when the CLT team is not present How often will refresher trainings be carried out, who has to attend?

Training of teachers is key in contributing to better learning outcomes of students in a long run. The more teachers are empowered in delivering digital content, the better for the students Execution of Digital Content using eShala The setup consists of a projector, a projector screen and a blackboard. The teachers go through the lesson plans using the tablets provided through the project. They study the content and plan on how to execute it. The Mini-PCs attached to the projector have the class content loaded allowing teachers to start up the projector and choose the lesson for the day. ●

This set up allows all students in class to watch videos and content projected on the screen. The teacher plays videos and explains additional concepts on the blackboard. One major concern repeated by most of the teachers interviewed is that the volume from the videos is very low and thus most students are not able to fully grasp what is being viewed. This set up also brings a few more questions that schools need to be keen on. 12

Comprehensive Guide for eShala DEP Social Impact Assessment

All students have different learning levels, how do teachers ensure that ALL of them are on the same page by the end of the class? How do teachers measure that learning is happening in real time? - Some students are shy to put up their hands and ask for the video to be played again. Do teachers encourage students to ask questions to measure their knowledge retention? How do assessments after each lesson help teachers plan better for the next class?

Unfortunately without observing an actual classroom implementation, it is difficult to come up with answers to these questions. SELCO and CLT India needs a mechanism in place that helps monitor the above and make sure learning happens the way it is supposed to.

FIgure 2: Projected content from the Mini-PC attached on the projector

Security and Safety of equipment Safety and security of the equipment is key when looking at long term impact. If the equipment is well taken care of, it increases its longevity and impacts many children’s lives. The schools are responsible for the safety of the equipment and to ensure that the equipment is in good working conditions, this involves not exposing the equipment to dust, harsh conditions and even petty thieves. ●

Both schools visited have a guard at the school who is responsible for guarding the school premises at night. One school even has a CCTV camera setup to monitor the equipment. This covers a portion of security concerns. Computer systems in practice need to be secured in conducive atmosphere, they are not supposed to exposed to dust and need to be serviced after a while to ensure that all inner components are well kept and functional. The eShala DEP setup is a fixed installation that makes for cleaning of the equipment relatively difficult. Also since the setup is fixed, it can not be uninstalled each day and kept away, that makes the equipment exposed to dust from daily classroom cleaning. 13

Comprehensive Guide for eShala DEP Social Impact Assessment

One of the schools I visited is improvising by using a dark cloth to cover the equipment to protect it from dust. While this is good effort, a better protective mechanism has to be considered by each school. Each school needs to invest in a protective case with a lock for the setup, this not only helps with the dust, but it also helps with security concerns should thieves break in.

Figure 6: A teacher covering the multimedia setup as a protection mechanism from dust

Content providers analysis (CLT India) CLT India provide the content used with the eShala DEP project. All of their content is pre-loaded on the Mini-PCs and they are the only ones supposed to perform an update of the content. The teachers raised some concerns regarding the content. Not all of the entire Syllabi is loaded and this means some lessons have to be taught in a traditional way without aid of the digital setup. ●

More concerns from teachers show that they feel there’s need for more diagrams and animations. The sound of most videos is low which is a big concern because then students are not able to comprehend what they watch in videos to be able to understand better and this can in the long run hinder their intended learning outcomes. Some of the equipment used in the video experiments in science are not easily accessible by students and so it makes it hard for the students to re-do the experiments back home or in the school labs. Content is king, and how you deliver it is even more important. The role of CLT India is a very crucial role in the success of effective implementation of the eShala DEP project. In these early stages, it is fundamental for CLT to put in place mechanisms to gather feedback from implementing schools to help them improve on their product.

Comprehensive Guide for eShala DEP Social Impact Assessment

There was no mechanism in place in the two schools I visited for which CLT India is using to collect feedback from teachers. It is not even clear how often CLT visits the schools to monitor or fix issues regarding content. CLT needs the following in place in order to prioritise their effective implementation for better learning outcomes; - A clear feedback mechanism that allows teachers to report content related issues - A timeline on providing feedback and answers to raised queries - Plan regular visits to schools to observe teachers as they conduct classes to see if they can pick up on some issues - Volume issues need to be looked into as soon as possible ● Equipment physical setup SELCO does a great job at installing the equipment. The equipment is installed at sealing level and allowed to hang a few meters to allow teachers to operate it. (See, figure 1: for reference). The setup design works well and allows for better security and limited disruptions.

However when it comes to science experiments, most teachers and students prefer the setup to be more portable to allow them transfer the setup to a lab where they can easily work on the experiments using the video references played. Because the setup is fixed, performing experiments from another location becomes a challenge. Should location of the equipment be well thought through before installation? The answer is yes. If the best place to put the setup is the school lab, each school has to weigh the pros and cons of having a setup in the lab. Better still, SELCO could come up with a different design setup where the equipment can be transferred to the lab for experiments and then back to the smart class.

Comprehensive Guide for eShala DEP Social Impact Assessment

Creating Impact Assessment Metrics Following the recommendations in the previous chapter, the cornerstone of CLT and SELCO’s impact lies in improving metrics. Right now, there no measures in place to help track the necessary metrics that lead to better project outcomes. Measuring the right impact outcome is key to the success of any social project or program. Let’s refer to Figure 4 below;

4.1 An outlook on Outputs, What gets measured? Outputs are the direct measurable results from SELCO’s installations. It is important to measure these because they help map out the outcomes and then overall impact. ● Number of eShala DEP installations; Measuring the number of installations leads to mapping the bigger impact SELCO makes in low income schools. Each installation means more students empowered with access to digital content, which in the long term helps with their learning outcomes.

Number of students trained; Each student trained using multimedia gains valuable skills that help them directly with improving their grades and retention of data. ●

Number of trained teachers; The use of digital tools in a class puts a teacher at a higher level. Through this program, teachers get trained in the use of multimedia and thus getting empowered to be better at their jobs. It is essential to capture this impact as an output of the project. ●

Comprehensive Guide for eShala DEP Social Impact Assessment

7: - An overview of the theory of change and output indicators Figure

Figure 8: Mapping the stakeholders affected by the project for each output indicator

Comprehensive Guide for eShala DEP Social Impact Assessment

4.2 Measuring eShala DEP Outcomes The outcomes are the specific changes in attitude, behaviour and skills obtained during the eShala DEP project. SELCO is keen on achieving the following outcomes as part of the project. -

Improvement in skills development of students Improvement in grades attained in subjected taught using multimedia (retention of knowledge) Improvement in the quality of classroom education Knowledge about sustainable energy and it’s importance Decreased Dropouts

For SELCO to effectively measure these, a few mechanisms have to be put in place in each implementing school. ➔ Assessments: In order to measure improvement in grades and retention of data obtained from the multimedia classes, a series of assessments have to be taken to help measure each student’s level of academic knowledge in subjects where smart class is used. It is essential to measure each child’s knowledge level at the beginning of term (BOT), mid term (MT) and end of term (EOT). These assessments can help in showing a progressive report of each individual student and can help in coming up with conclusions of each student’s learning outcomes.

Comprehensive Guide for eShala DEP Social Impact Assessment

➔ Capture teachers & students testimonies: Regularly capture testimonies of teachers and students who use the smart class. Testimonies can be captured using video or any other digital media. Telling the story in the beneficiaries eyes is key in communicating impact. Here are some of the questions teachers can answer; -

How do teachers feel empowered? What has changed for them since using digital content? How do they see this way of teaching improving their student’s outcomes?

The same can go for students, in addition to getting their testimonies, compare their grades with other students from different schools who are not using smart class. How do these results vary? Can the data obtained help come up with conclusions about the use of multimedia? ➔ Assess knowledge on sustainable energy practices: For every student before using smart class, take their knowledge level about renewable energy and capture findings in each student file. Revisit this file at the end of term and ask them about solar and renewable energy after they have been exposed to smart class and the SELCO solar installation. How much data do they now know? Can they name other sustainable energy practices? ➔ Measure soft skills: Using smart class exposes a lot to students, the nature of instruction allows for students to be more engaging than traditional classes. Find ways of measuring students behaviour changes in a smart class. Key areas to look out for; - Are students more excited about class? What are the average attendance levels compared to other classes without smart class? - How involved are students in class? Do they ask more questions than before? - Are the students more curious to find out more on their own? - What projects have students undertaken on their own? Noticing and documenting these soft skills is fundamental in helping follow up each student's learning curve. Through measuring the above outcomes, it becomes relatively easy to know if the program is achieving its intended objectives or if some things have to be tweaked to help achieve impact. Teachers and School administration need to be trained on capturing the various data points to help analyse learning outcomes and impact.

Comprehensive Guide for eShala DEP Social Impact Assessment

Conclusion Measuring the right metrics is the cornerstone of every project’s success. For the DEP project, ensuring every stakeholder knows what and how they are measuring their impact will enable the program to impact many children in low income schools. SELCO needs to align its goals with CLT India and develop tangible metrics that each of them can measure on a regular basis to ensure project outcomes are met.

Comprehensive Guide for eShala DEP Social Impact Assessment

Abbreviations SELCO - Key implementing partner of the eShala DEP project DEP - Digital Education Program eShala - Name of the Digital Education Program implemented by SELCO CLT India - Official content partners of the Digital Education Program BOT - Beginning of Term MT - Mid Term EOT - End of Term

Comprehensive Guide for eShala DEP Social Impact Assessment

Appendix: Stakeholder questionnaires

Comprehensive Guide for eShala DEP Social Impact Assessment