What do they know? A summary of India's National Achievement Survey, Class V, Cycle 3, 2010/11

Introduction

The 2010/11 National Achievement Survey (NAS) of Class V students has been significantly updated in line with international best practice in student assessment. Conducted by the National Council of Educational Research and Training (NCERT), this marks the most comprehensive educational assessment survey in India to date.

Between March and May 2010, the survey tested over one hundred thousand students in language, mathematics and environmental studies. In addition, questionnaires were used to collect information on schools, teachers, and pupils’ school experiences and home backgrounds.

This booklet summarises the NAS methodology and findings in order to:

• explain improvements made in assessment techniques

• provide a snapshot of achievement levels in Class V

• highlight areas for further investigation

Supporting the National Council of Educational Research and Training

The UK government-funded Technical Cooperation Fund is supporting NCERT to reach international professional standards in National Achievement Surveys.

To achieve this, technical assistance has been provided in the following four areas:

• Test development and psychometric analysis.

Theory, research, statistics and international experience relating to learning assessment systems

•

• Data management and reportin

Administering, processing and analysing large-scale assessments

• g

States of change – why we evaluate

India has made a significant investment in its education. The government’s flagship programme Sarva Shiksha Abhiyan (SSA) is designed to ensure access, equity and quality in elementary education. The nation now needs reliable information about students’ achievement in order to judge the quality of education provided.

The history of NAS

Carried out as part of SSA, the NAS aims to collect reliable information about the achievement levels of students in government and government-aided elementary schools.

In 2000, NCERT’s NAS programme was incorporated into the SSA programme. The plan was to carry out three NAS cycles, each cycle covering three key grades: Class III, • Class V, • Class VII/VIII.

All three Classes are tested in mathematics and language. In Class V, students are also tested in environmental studies (EVS), while Class VII/VIII completes tests in science and social science.

The Baseline Achievement Survey (BAS) was carried out in 2001-2004, followed by the Midterm Achievement Survey (MAS) in 2005-2008. The experience gained through these initial two cycles made the value of the NAS clear, and the surveys were made an ongoing feature of the national education system. To mark this shift from stand-alone surveys to continuous assessment, the Terminal Achievement Survey (TAS) has been renamed ‘Cycle 3’.

Measuring progress in education

The NAS is a useful tool for teachers and policymakers alike – to establish what students are achieving in core subjects and to identify any areas of concern. By repeating the NAS at regular intervals, the data can be used to measure trends in education achievement levels and measure progress made by SSA and other education reforms.

Sarva Shiksha Abhiyan – towards universal elementary education

The 86th amendment to the Constitution of India made free and compulsory education for children aged 6-14 a fundamental right. The government’s SSA programme is providing support to achieve this goal.

Spanning the whole country the programme addresses the needs of nearly 200 million children. In areas with no or inadequate schooling facilities, SSA is building new schools or improving existing infrastructure. Teaching capacity is being improved through recruitment and extensive teacher training and many other inputs.

See website for more details: www.ssa.nic.in

Turning test results into trends – methodology

For the first time in India, the Class V NAS has established robust baselines for achievement in reading comprehension, mathematics and environmental studies. The introduction of sophisticated, internationally-recognised assessment methods means that future trends in educational standards can be judged with confidence against these new baselines.

Developing instruments is a crucial activity in any survey. In preparation for NAS Class V, NCERT developed test booklets in language, maths and environmental studies. In addition, questionnaires were designed for students, teachers and schools to capture information on school environment, teacher qualifications and status, and student background.

Developing Class V tests

The Class V NAS differs from previous cycles of the survey as it uses Item Response Theory (IRT) to place results on meaningful scales, which are independent of the particular tests used. This method is in line with international best practice.

What is IRT?

Item Response Theory, or IRT, uses a mathematical model to calculate the probability that a student will respond correctly to a specific test question, based on the student’s level of ability and the degree of difficulty of the questions concerned.

Previous Class V survey results were reported as the percentage of correct answers. This meant that results were specific to that year’s test and could not be compared to results from other tests or survey cycles. The scales created using IRT, however, permit future comparison in each subject.

For each of the three subjects, the Class V syllabus and textbooks from across the country were used to develop assessment frameworks, which described the knowledge and skills to be covered in the tests.

Once developed, the test booklets were translated into 15 regional languages. The tests were then piloted in 21 states to assess how the items worked across the different languages.

Unlike previous surveys, this round of the Class V survey used three test booklets per subject rather than one. This allowed assessment in greater depth. To enable comparison across all three tests, a block of common questions or ‘anchor items’ was used in all booklets.

For each subject the NAS presents ‘item maps’ which arrange items according to their level of difficulty. The scale score of an item on the map indicates the ability level of students who, on average, are likely to answer it correctly. Hence, the maps can be used to describe what students at different levels of ability know and can do in each subject.

Selecting samples and participants

Cycle 3 of NAS Class V used tests and questionnaires to gather information from 122,543 students in 6,602 schools across 31 states and union territories (UTs).

In each state and UT, the sample was generally selected in three stages. Firstly, districts were selected based on the number of Class V students enrolled. Secondly, the requisite number of schools was selected. Finally, the required number of Class V students in the school was selected at random.

This summary document concentrates on results from 20 states where the sample size was higher than 80% and students were tested while still in Class V. In some states, low coverage and delays in carrying out tests mean that the results should be read with caution.

Analysing the results

Following IRT analysis, the Class V test results have been placed on a scale, which was normalised with a mid-point of 250. Achievement scores on this scale cannot be interpreted as being equivalent to any particular score on any particular test. Instead, they can be used to make meaningful comparisons between states. The results can identify relatively high achieving states and those states in which achievement is relatively weak.

Not only does the survey report the average performance in each state, it also presents the range of students’ performance across the ability range. This range of achievement is presented as five key ‘percentiles’: 10th, 25th, 50th, 75th and 90th. For example, the score at the 25th percentile is that which 75% of students achieve or surpass, and the score at the 90th percentile is that which 10% of students achieve or surpass.

The range between the 25th and 75th percentile, or the ‘inter-quartile range’, represents the middle 50% of students. The smaller the inter-quartile range, the greater the homogeneity within the state’s student cohort. This allows us to make judgements about issues of equity.

Snapshot of a nation – the results

Overall the NAS reveals significant differences in the average achievement levels of students between states. Some difference may be accounted for by contextual factors, but on balance the results suggest that the quality of educational outcomes is far from equal across the country.

The survey’s findings in reading comprehension and mathematics are of particular concern. While the highest achieving students can read with understanding and can successfully solve more complex mathematical problems, the bottom 25% of students appear to function at low levels of literacy and numeracy. For example, they cannot read with understanding if the context is unclear and they cannot solve calculations involving basic mathematical operations.

NAS results show great diversity in achievement between the highest and lowest performing students in some states. Inequity of outcomes suggests that, in some states, more able students are receiving support while their less able peers may not be enjoying adequate opportunities to reach more acceptable levels of learning.

National trends

The NAS found the following national trends in Class V students:

There was no significant difference between boys and girls achievement levels in reading, maths and environmental studies.

There was no significant difference in achievement between urban and rural students.

But… Students from minority groups, comprising Scheduled Castes, Scheduled Tribes and Other Backward Categories, scored significantly lower than students in the General category

Achievement by subject: Reading

State's average score is not signi cantly di erent to group average State's average score is signi cantly above group average State's average score is signi cantly below group average Not repor ted

Andaman & Nicobar Islands

Chandigarh

Delhi

Puducherry

Tripura

Rajasthan

Haryana

Gujarat

Punjab

Himachal Pradesh

Uttar Pradesh

Andhra Pradesh

Bihar

Tamil Nadu

Karnataka

Madhya Pradesh

Jammu & Kashmir

Uttarakhand

Orissa

Chhattisgarh

The language exam was designed to test a range of reading skills: locating information, identifying and interpreting ideas, and demonstrating understanding of information that may not be explicit in the text.

There was a great variation in students’ ability in reading comprehension. At the lower end of the scale, about 65% of students were able to locate information given in a table.

Intermediate students could do more, for example, around 50% could identify the cause and frequency of an event, and draw conclusions about characters from information given in the text. The relatively small proportion of students, approximately 32%, at the higher end of the scale could read with understanding and infer things alluded to, but not given explicitly in the text.

Achievement by subject: Mathematics

Maths

State's average score is not signi cantly di erent to group average State's average score is signi cantly above group average State's average score is signi cantly below group average Not repor ted

Uttar Pradesh

Andhra Pradesh

Bihar

Rajasthan

Haryana

Chandigarh

Andaman & Nicobar Islands

Delhi

Tamil Nadu

Gujarat

Jammu & Kashmir

Karnataka

Madhya Pradesh

Puducherry

Punjab

Tripura

Himachal Pradesh

Uttarakhand

Orissa

Chhattisgarh

Mathematics

The Class V mathematics exam tested students’ achievement levels in arithmetic, algebra and geometry. In all three domains, students demonstrated a very wide range of ability.

About three-quarters of students successfully found the answer to the sum ‘7010 + 2699’. However, fewer than half could answer the question ‘How much greater is 555 than 198?’

Disappointingly, only about a third of students could find the difference between two numbers given in decimal notation – even when four alternative values were given.

7010 + 2699 = ?

Achievement by subject: Environmental studies EVS

State's average score is not signi cantly di erent to group average State's average score is signi cantly above group average State's average score is signi cantly below group average Not repor ted

Chandigarh

Delhi

Puducherry

Tripura

Rajasthan

Haryana

Gujarat

Punjab

Himachal Pradesh

Uttar Pradesh

Andhra Pradesh

Bihar

Tamil Nadu

Karnataka

Madhya Pradesh

Andaman & Nicobar Islands

Jammu & Kashmir

Uttarakhand

Orissa

Chhattisgarh

Environmental studies

The environmental studies (EVS) exam tested students’ knowledge of social studies and science.

In EVS, students of average ability and above could extract information from a basic bar-chart and could interpret a simple family tree. However, only about a half of students appeared to know that the part of a carrot we eat is, in fact, the root. And only a third of the students successfully identified energy sources and their by-products.

Background factors – survey results

The NAS survey used three questionnaires designed for schools, teachers and pupils, to collect information on background factors that could potentially influence learning outcomes.

The results suggest that coming from a larger family is related to lower than average performance. Students with only one sibling outperform by a small, but statistically significant amount, those who have two or more brothers and sisters. This effect is lowered, but does not disappear entirely, when key factors, including socio-economic background, are accounted for.

Incentive schemes such as free uniforms and full scholarships appear to be related to higher achievement. However, since these interventions target disadvantaged groups rather than the whole student population, these findings should be treated with some caution until further investigation is carried out.

Students who are given homework every night and have it checked by their teacher do better than their peers. Perhaps surprisingly, after allowing for key background variables, pupils who have a tutor do worse than those who do not. One plausible explanation for this is that families are more likely to hire a tutor if their child is already struggling.

Activities such as watching TV, ‘hanging out’ with friends, using the internet and doing homework all appear to be related to higher attainment. This effect is reduced after allowing for the key variables, suggesting that this may be due, to some extent, to the availability of resources. ‘Playing on the computer’ is not associated with higher attainment and, after allowing for the key background variables, appears to have a negative impact on achievement levels.

Nearly 90% of the students who took part in the Class V survey reported that they like being in school

Data is just the start – what next?

The Class V NAS is important in its own right, but it cannot answer every question on the quality of India’s elementary education.

National Achievement Surveys provide reliable information about levels of educational outcome and also give quantitative data about the relationships between background variables and outcomes. This objective information is an important starting point for identifying challenges within teaching and learning and deciding on the best way to improve educational processes. For example, NAS results can identify the questions that students at different levels of ability find most difficult. It is then for curriculum and teaching specialists to decide how such difficulties can be resolved by, for example, changing teaching practice and improving textbooks and other materials.

Making the most of the results

Each NAS cycle should be followed by further analysis and action by national and state level decision makers. For example, state educational authorities should review findings within their local context and against local priorities. It is recommended that state-specific reports and corresponding action plans are produced.

The diagram below shows how data from NAS is the starting point in a continuous cycle of quality improvement in education.

Ongoing improvements

The Class V Cycle 3 NAS was more sophisticated than earlier surveys, however conducting large-scale assessments in India remains technically challenging.

Three lessons learnt during this latest survey offer the potential to improve future NAS cycles:

Increasing the quality of quantitative information

• about schools and enrolment will help eliminate current difficulties in drawing truly representative samples. Investment in national and local education information management systems would be of great benefit.

Development of future survey tests should

• be based on a national debate about which curriculum domains should be given priority –should surveys focus on subject curricula or on a limited number of critical competencies such as numeracy and reading literacy?

• Surveys the findings and implications need to be communicated to a wider public, not just education specialists.

To maximise the value of National Achievement

Find out more:

The full NAS Class V report will be available online. To read this and other evaluation reports relating to the SSA, please visit the SSA-TCF website: www.ssatcfund.org

National Assessment