CIES 2015: India's National Achievement Survey

Working with NCERT on the Government of India’s National Achievement Survey Jayshree Oza

Executive Summary

This paper describes in summary the progressive development of the National Achievement Surveys (NAS) of the Government of India, with the parallel evolution of technical support provided to it by Cambridge Education under two national education programmes: Sarva Shiksha Abhiyan (SSA), and Rashtriya Madhyamik Shiksha Abhiyan (RMSA). The first is India’s flagship programme for universalisation of elementary education. The second is a newer programme to enhance the quality and equity of secondary education.

This narrative illustrates how medium term international technical assistance, operating through the period 2008 to 2016, on a relatively modest scale, can make a sustainable longterm difference to a large scale education system.

1 History of the technical cooperation development

Cambridge Education is an international consultancy company. It was awarded two technical cooperation contracts under open competitive international tender by the British Government’s Department for Overseas Development (DFID), to provide technical assistance and capacity development support services to the Government of India, in specified parts of the education system, first for elementary education then for secondary education.

Under the first project, the technical assistance was contained entirely within one institution, the National Council of Educational Research and Training (NCERT), and focused on building capacity in only two areas: programme evaluation and national achievement surveys. In the second project, the support covers a broader range of technical areas, and direct engagement with a wider range of institutions, ie, the Ministry of Human Resource Development, NCERT, the National University of Educational Planning and Administration (NUEPA), and the five Regional Institutes of Education across the country.

2 Working within Government of India institutions and systems

Cambridge Education leads a consortium whose unified identity to carry out its functions supporting RMSA is as the Technical Cooperation Agency (TCA). The TCA has established teams in five regional centres, based within the Regional Institutes of Education which, along with the central National Institute of Education, make up NCERT, and aims to serve 37 states and Union Territories of India.

The TCA provides technical services of guiding policy, planning and monitoring as well as capacity building across a wide range of areas: Teacher Management and Development, School Quality, Data Management and Use, Results-Focused Planning, and Communications and Knowledge Management. National Achievement Survey (NAS) is one of the six thematic areas of support that reaches across India.

3 Technical innovation through the TA: shift from the old NAS to the new NAS

India is a unique mix of diversity and shared culture and histories. The Indian education system includes students studying in different languages, in diverse geographical, economical and socio-cultural conditions. Each state runs its own education system, within a national policy framework.

There has been large commitment of resources to education, by states, federal government and parents. However, the Government, development partners and the general public, have been concerned that the education system is not serving the learning needs of all its students. Hard evidence about what is working and what is not working has been lacking.

The National Achievement Survey was therefore initiated by the Government of India. The Survey is tasked to find out what students in different grades know and can do and to paint an informative picture of educational achievement across the range of the school system.

The purpose of the National Achievement Survey is to assess and improve the system. The Government of India had implemented one National Achievement Survey prior to the commencement of technical assistance, using traditional methods based on Classical Test Theory. The Government requested the initial technical assistance under SSA to help develop and strengthen the capacity of the government

agency charged with undertaking the NAS, the National Council of Educational Research and Training (NCERT), specifically requesting that the NAS move to utilisation of an approach based on Item Response Theory (IRT). Using modern assessment survey techniques, far richer information is available by this survey method than from former survey methods or from traditional tests and examinations.

The adoption of techniques based on Item Response Theory has therefore been central to the technical support and the development of NCERT’s NAS through this technical assistance relationship. IRT is the statistical approach used in leading international surveys such as PISA, and in the Nation’s Report Card, NAEP, in USA. A large proportion of the technical assistance provided to the National Achievement Survey is in learning and applying elements of IRT methods.

Item Response Theory enables tests to be designed and results to be scaled so that the true ability of the learners who sit the test is measured, even if the test items (or questions) differ; the performance of learners who sit tests in different languages and in different locations, on different dates, can be compared accurately; and accurate comparison of performance can be made of one year group of learners on subsequent occasions, when different learners are populating that year group.

Technical assistance is providing advice, training and support to the Government organisation responsible for conducting the surveys. Thus NCERT is becoming stronger in programming and managing the Surveys; the technical capabilities of individual staff are being developed to undertake each technical step of the process of developing, trialling, carrying out, analysing and reporting the National Achievement Survey at different Grade levels.

4 The design of NAS as an instrument to promote equity

The design of the National Achievement Survey is aligned with increasing equity, inclusion and quality of education. This is manifest in a range of dimensions:

° The NAS reporting focuses on equity issues by providing information on learning achievements disaggregated in terms of State, gender, location (rural/urban), and by social group, including identified disadvantaged groups (eg, Scheduled Tribe, Scheduled Caste, Other Backward Caste).

° NAS operates at a range of school Grade levels (Classes 3, 5, and 8 at elementary level, and Class 10 at secondary level). Thus it captures information about progress and achievement at different levels of the school system to address issues that are identified in the system through this process.

° Besides the subject-wise tests of students, questionnaires are administered to understand the background factors that influence student performance, such as the qualifications and experience of teachers, characteristics of schools and home conditions.

° Possible policy interventions are identified through consideration of the NAS findings to strengthen the conditions for learning, and, where problems are identified, to mitigate disadvantage and to build social capital for the disadvantaged students.

° The findings of the report may be used by various stakeholders, eg, policy makers, curriculum developers, trainers and other education practitioners such as managers, administrators and teachers.

° Whilst the NAS reports levels of achievement by State, and it does report on performance in each State in comparison to national averages, it avoids supporting the development of a culture of school ‘league tables’, which could lead to more inequality.

° By using scientific sampling and IRT methods, misleading correlations can be avoided, whilst a reliable picture of achievement can be generated.

° The impact of the NAS reports in their ability to inform and influence planning and implementation at all levels (from local levels, through State to National level) is increased by the transition from the former approach of publication of a single, full technical report, towards more rapid production and wider dissemination of transparent, readily accessible reports, using infographics and summary formats to pick out significant highlights, as well as the full technical reports, and making these available online as well as in paper form.

5 Challenges in operating in this way

There are substantial challenges which face the Government of India and its technical assistance partners in developing and institutionalising NAS, the innovations of IRT, and enhanced, user-friendly reporting, despite the wholehearted commitment and unified desire of all parties involved to do so.

° A very small team attempts to shoulder the core work of a nationwide survey; and team members are often drawn away to attend to other responsibilities. Each organisational team member has to cover a range of roles on different surveys taking place concurrently.

° The team is learning radically new methods which can be technically very challenging to learn for immediate implementation.

° The institutional and political pressures and expectations of dramatic change and producing rapid results far outstrip the organisational capacity to deliver them.

° Institutional norms are sometimes at variance with externally introduced technical ideas, requiring negotiation and adjustments.

° The timeframes for technical assistance and for the cycles of survey development and reporting are very short.

In combination, these factors pose major obstacles to overcome and constrain the rapid and thorough adoption of the new National Achievement Survey.

6 What the model of technical assistance demonstrates

This experience demonstrates that it is possible to have a controlled, phased progression from a technical cooperation relationship which is based on high reliance on external technical support, to one where technical capacity has been more established in the beneficiary, with the external technical support still assisting, but without creating dependency. Throughout the process, leadership and ownership of the surveys, and the decision-making authority, have remained firmly in the hands of the Government of India.

This has been the case from the earliest stages, when all the technical skills and knowledge about IRT and related survey assessment methods and procedures were completely new territory for the NCERT personnel involved in developing and running NAS, through the interim learning process, and into the later phase where NCERT is managing and carrying out all the technical tasks, with the TCA assisting and providing further technical training, to strengthen and entrench the expertise in NCERT.

In conclusion, aside from the technical and situational specifics, the capacity building taking place in the Indian National Achievement Survey illustrates more general points about using technical assistance to leverage greater equity and inclusion in education on a large scale.

° Firstly, for a relatively small investment, it is possible to generate substantial positive change through technical assistance.

° Secondly, sustainable development can result from working with government - by developing the capacity of its institutions; by providing world class technical support; and by not supplanting the government’s role by a crude approach of short term substitution of skills.

° Thirdly, the experience confirms what has long been understood in development work: that lasting change is not an overnight success through rapid external intervention: planning needs to be for the longer period, beyond short term technical assistance

For further information, please visit www.rmsaindia.org

To contact the TCA supporting NAS, please email jayshree.oza@rmsatca.org

The TCA is led by Cambridge Education