UKFIET 2015: Early grade literacy assessment and language instruction in Liberia

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

Tools to assess early literacy skills are broadly used by development partners to assess the effectiveness of interventions. In particular, the Early Grade Reading Assessment (EGRA) tool is currently applied in more than 60 countries. The value placed on EGRA sub-tests can often be seen in pedagogical responses to the assessment, with a heavy focus on phonological awareness, phonics and reading fluency. However, EGRA tools are not as effective in measuring oral language development skills including receptive and expressive vocabulary knowledge. This paper describes the development, field testing and verification of a new, expanded version of EGRA, named the Early Grade Literacy Assessment (EGLA). The EGLA was developed by Concern Worldwide in Liberia in Bassa and English in 2014. EGLA maintains core sub-tests from EGRA, and includes new measures for receptive and expressive vocabulary, verified against a norm-referenced picturebased vocabulary test.

In this presentation, an adapted vocabulary measure, the Two-Question Vocabulary Test is presented. The measure can be group administered and is appropriate for children in the early primary grades. The validity and reliability of the measure is provided, including a rationale for its use with children in Liberia. We will explain the adaptation procedures during the development process, lessons learned, and procedures for adaptation.

Finally, the implications of these findings on literacy interventions in Liberia will be presented. The inclusion of expressive and receptive language skills in assessments in Liberia provides opportunities to expand the scope of teacher training on language instruction in both mother-tongue and English as a second language. Recommendations are provided to inform further expansion of reading assessments to include vocabulary and influence improved pedagogy of language.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

1 Introduction

Globally, 250 million children of primary school age, whether they are in school or not, lack basic reading and writing skills (Education for All, Global Monitoring Report, 2012). In addition, in some countries, the increase in literacy rates in recent years has begun to stagnate, although we now know how to teach young children to read efficiently and effectively in multiple local languages (Pratham, 2015). There is considerable evidence for the need for coaching support to teachers as they implement new and more effective reading programs in the primary grades, yet we remain a long way from reaching the 2015 global goal of cutting illiteracy in half.

The process of improving the literacy rates in the poorest countries often begins with a national assessment of student literacy skills. One of the most common global literacy assessments administered to children in the primary grades in developing countries is the Early Grade Reading Assessment (RTI International, 2009), a valid and reliable measure in which children are assessed individually on items that reflect foundational early skills in reading acquisition. Another effective approach to measuring the reading skills of young children is the ASER assessment, the annual Status of Education Report from Pratham, India (ASER Centre, 2015) and its Eastern Africa version, UWEZO. This is a household survey in which children are assessed in their homes capturing data on literacy from children who attend school as well as from those who do not. These tasks are simple assessments that tap into the early literacy skills of young children who should be learning how to read in school. Both of these assessments reveal that many young children in poor countries –sometimes more than 90% - are, in fact, not learning to read by the end of grade 3, whether they are attending school or not (e.g., Stark, 2011; Muralhidaran, 2013; ).

The current systems level assessments provide important information that guides decisions about interventions to improve the quality of education in the primary grades. Current USAID, DFID, World Bank, and other multi-national donors fund projects from small scale projects in local communities to large national scale projects focused on improving literacy levels of primary grade children in thousands of schools. Many of these projects have collected data that reflect significant improvements in literacy skills, but in even the most successful projects, many of the students are not literate after 2 years of intervention. An important challenge is to determine how to improve interventions so that more students reach reasonable levels of reading fluency with comprehension after two years of intervention. An increased emphasis on developing language comprehension skills, including vocabulary is one possible avenue to increase learning levels.

Children in current high quality reading programs in the least developed countries (LDCs) may be learning to decode and to read words with adequate speed and accuracy, but they may not be developing the necessary vocabulary and comprehension skills they need to move from reading at a basic level to reading proficiently. Most early reading intervention programs focus on teaching children to speak and understand the official language (L2) as they are being taught to read in their local language (L1). But few programs examine whether students have adequate vocabulary and oral comprehension in the language that they speak and understand.

The Simple View of Reading model reminds us that learning to decode is necessary but not sufficient to developing reading proficiency. Language comprehension is also critical to becoming a good reader. The emphasis in most programs is on developing speaking and listening language skills in the official language (L2), with an assumption that the mother tongue/local language (L1) oral language skills are sufficient. But research on oral language development suggests that this assumption may not be warranted.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

Whatever the language of literacy, solid foundations in oral language, particularly vocabulary knowledge and sentence comprehension, are essential prerequisites for literacy development in that language. It should be emphasised that there are benefits irrespective of whether these foundations are in the ‘standard’ language or non-standard varieties; the crucial point is that children should have sufficient mastery of vocabulary and syntax to enable them to make inferences from text. (Nag & Snowling, 2014, p. 2)

In the Simple View, the basic tool skills in learning to read, such as phonological awareness, learning to sound out and read words can be considered the “skills-based competencies” (Lesaux, 2012). But an equally important component of learning to read lies in “knowledge-based competencies” (Lesaux, 2012) that include vocabulary and conceptual knowledge required to make meaning of what is being read. Lesaux draws a distinction between the “skills-based competencies” that readers need to sound out and recognize words and the “knowledge-based competencies” that include the conceptual and vocabulary knowledge necessary to comprehend a text’s meaning. The competencies that depend upon vocabulary and conceptual knowledge contribute significantly to lasting differences among children in reading proficiency over time, particularly for those children who grow up in very poor homes with few to no books. Thus, it is imperative that vocabulary and language concepts be taught early and effectively. The consequences for not teaching these competencies may not be evident until the middle years of school, but they ultimately contribute the most to meeting the goal of being a truly literate adult.

There is also some evidence that in transparent languages, oral comprehension is a strong predictor of reading comprehension for primary grade students with reading accuracy providing a significant but minor role (Tobia & Bonifacci, 2015). Reading speed continues to be considered perhaps the strongest predictor of reading comprehension according to a recent meta-analysis (Florit & Cain, 2011).

Because intervention programs tend to teach what is tested, the content of many early grade reading programs address the skills that are tested in the EGRA, UWEZO, ASER and similar assessments. That means the emphasis is on symbol knowledge and phonological awareness, including syllable awareness as well as phoneme awareness as core tool skills that will facilitate the process of developing fluency in reading. While these assessments tap critically important skills in early reading acquisition, they cannot provide sufficient information on L1 or L2 language/vocabulary skills of students to inform policymakers and curriculum developers on what language skills might be needed. Without data on vocabulary knowledge of children in the early primary years, it is difficult to identify whether the gaps in vocabulary and language comprehension are significant or whether current programs are improving vocabulary and language skills.

There is a broad research consensus (e.g., Biemiller, 2003; Cain & Oakhill, 2003; Scarborough, 2001; Whitehurst & Lonigan, 2001) that by the time students reach middle to secondary level schooling; oral language/vocabulary knowledge is a stronger predictor of academic success than decoding skills. In fact, as students become more proficient in decoding (to the point that nearly all words are recognized instantly and automatically), the role of vocabulary increases as students become better readers. Despite the importance of vocabulary and oral language skills in the development of proficient reading, there are few primary grade assessment tasks that directly tap vocabulary knowledge in early grade reading assessments in LDCs. Most programs include some instruction in new vocabulary skills that are based upon current research on effective methods for vocabulary instruction in English, but there are few resources available to guide program developers in addressing local language vocabulary instruction and effective approaches to teaching relevant skills in morphological awareness, syntax and grammar.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

2 Assessing beginning literacy skills

Recent national assessment reports, including EGRA, UWEZO, and ASER assessment data, remind governments and stakeholders of the critical importance of continuing and even increasing the efforts to improve the learning outcomes for children in the primary grades in the poorest countries. The UWEZO, ASER and EGRA assessments provide important systems-level data on the status of student literacy attainment in the early grades. Prior to the development of these assessment tools, the only data on student achievement in many countries was obtained from the standardised tests administered at the end of the primary grade cycle at the 6th grade, or mid-secondary school typically. Thus, until recently, no information on the progress of student learning was available for policy makers until it was too late to intervene. Both the ASER and UWEZO tests were created in the contexts in which they are most commonly utilized. The EGRA measure was developed in the U.S., but has been adapted for use in over 60 countries in multiple languages and cultural contexts. A closer look at these instruments and what they measure is useful for determining effective ways to supplement current assessment tools with language comprehension tasks.

ASER (2015): For the past 10 years, data have been collected by local citizens in India on whether children are enrolled in school and whether they are learning basic math and reading skills. The Annual Status of Education Report or ASER is the largest annual household survey carried out by citizens of India. While enrolment in India is high with 96.7% of children enrolled in school, national figures for rural India indicate that more than half of all children in Standard 5 cannot read a Standard 2 level text fluently. The ASER assessment in reading is organized so that a child begins with a story reading task and is directed to progressively easier tasks if unable to complete the first task. The tasks include a simple story that a child reads aloud and answers questions about, a short paragraph, a few sentences, a short list of words, and letters of the alphabet. There is no direct assessment of language comprehension beyond the simple reading tasks.

UWEZO: UWEZO assessments are conducted in 3 Eastern African Countries: Uganda, Tanzania, and Kenya. The assessment tools are similar to ASER tasks in that they are household surveys conducted in local villages by trained volunteer assessors and include reading and math items similar to those developed for ASER. The 2013 report summary indicates that only 25% of students in Standard 3 in the three UWEZO countries are able to read fluently.

The principal finding in this report, echoing results from previous years, is that children are not learning. Children are not acquiring foundational skills of literacy and numeracy consistent with the official curricular requirements in their countries. (UWEZO, 2013 assessment report)

EGRA: The EGRA assessment is individually orally administered to students ensuring that incorrect responses are not due to a student’s inability to read and understand a written test question. Children are asked to identify letters and letter sounds, discriminate beginning sounds in words, identify familiar words, decode non-real words, read a simple passage and answer questions about what they read. Sometimes, they are asked to write a sentence from dictation and to answer questions from a brief passage read aloud to them. Many of the tasks are timed which increases the reliability of the task and provides information on the automaticity of a child’s skill on that task. Data from these tasks provide an accurate impression on the general early reading skills of children in the primary grades and the measure has been adapted in many languages including both alphabetic scripts and non-alphabetic scripts. The results of assessments indicate that in many of the poorest countries, very few children can read any words or even name all of the letters of the alphabet by the end of grade 3.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

3 Why collecting valid information on children’s oral language skills in L1 and L2 is important

In these early grade national literacy assessments, it is quite evident that children are not learning to read at an acceptable rate. However, the data from the measures target skills in learning to read words and to understand grade-level text. There are no tasks that tap a child’s knowledge of vocabulary in the language of instruction. While an emphasis on word reading skills, comprehension, and fluency is appropriate and necessary, it is also important to teach children new vocabulary in their local language. But we have no data on levels of vocabulary of children as they begin school, thus, establishing a rationale for the importance of teaching vocabulary, and setting learning goals for children can be challenging.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

4 Efficient and useful language assessment for young children in developing countries

In the U.S., by 12th grade, students who began school with limited vocabulary knowledge may know half as many academically relevant words as students who began school with average vocabulary knowledge (Biemiller & Slonim, 2001; see Hart & Risley, 1995; White, Graves, & Slater, 1990).

While we do not have data on the levels of vocabulary knowledge in many countries, it is likely that the predictive validity of vocabulary knowledge has relevance for educational achievement universally. The definition of ‘academically relevant’ will change based upon the context, but the importance of strong vocabulary skills is universal. In order to ensure that sufficient vocabulary knowledge is developed in the mother tongue/local language, reading programs need to include an emphasis on building knowledge of word meanings at an oral level initially, followed by knowledge of written words as children learn to read. Vocabulary knowledge is often reflected in the quality of responses a child makes to story prompts, in the quality of vocabulary used in story recall and retells, and in the identification of the correct meanings in individual vocabulary words. However, story retell can be time consuming and requires expert knowledge in the scoring process. Correct word meanings require that a child read an item and respond in writing. Neither option is particularly feasible in impact evaluations among national samples of students in developing countries in which few children in the primary grades can read more than a few words. Furthermore, some of the widely used published vocabulary assessments are either lengthy one to one assessments such as the Peabody Picture Vocabulary Test and its international variations, or tests requiring children to read words and to respond in writing. There are few measures that are efficient, reliable in terms of cultural and linguistic relevance, and very easy to administer and score.

One option to consider is a parent questionnaire or survey. Parental questionnaires about language development can be very informative (Ireton & Glascoe, 1995). One example is the Child Development Inventory (Ireton, 1992). In this inventory, parents are asked questions about their child’s language, but these questions are quite general and were developed in the U.S. and Western Europe so would need to be adapted significantly for use in LDCs. Still, these parent interview measures are not direct measures of a child’s language skills. An advantage of a direct measure is to provide a baseline of performance and then to collect midline and end line data to determine the progress of a group of students in improving vocabulary knowledge over time.

The EGRA test is an important tool to identify student needs and determine the impact of classroom interventions. Since it is an orally administered test, an oral vocabulary measure might be a logical choice. There are challenges in developing an oral vocabulary measure that is designed to be flexible enough for adaption in multiple languages and cultures. There is a need, for example, to identify a useful list of vocabulary words in the local language that children in the early primary grades would be expected to know and understand. One example of a measure that has promise is the Two Questions Vocabulary Measure (Kearns & Biemiller, 2010).

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

5 The Two Questions Vocabulary Measure (TQVM): Development

The Two Questions Vocabulary Measure (Kearns & Biemiller, 2010) was developed to address the need for a language assessment tool that could be administered orally to groups of young children. The test was designed to use vocabulary words selected from the Peabody Picture Vocabulary Test –III (PPVT) and then to present the items orally to groups of students in kindergarten, grade 1 and grade 2. The sample included students whose first language was other than English. Students responded to the task by circling the correct response. All of the words in the sentences were designed to be familiar to children except for the tested word and the answers to each question required a one-word response: yes or no. For example, below is a sample item:

a) Does a cow give milk? (yes)

b) Can a cow run faster than a horse? (no)

The measure included 22 words selected from sets of vocabulary words from the Peabody Picture Vocabulary Test, III (citation). The words selected spanned an age range from age 4 through age 16 and gradually increased in difficulty. The measure was administered in a group setting with students responding with a happy face rebus followed by the word ‘yes’ and a frown face and ‘no’ as the two selections. They responded to the questions by circling one of the two options. Analysis revealed a correlation of .78 between the raw measures of the PPVT and TQVM. Scores obtained on the TQVM were lower for all grade levels tested than scores on the PPVT. That is to be expected since the TQVM taps a deeper level of vocabulary knowledge than the PPVT. Responses require that the respondent use the meaning of the word to answer the questions or to respond to the statements rather than identifying a correct illustration of the word meaning in a limited context, such as the PPVT.

The TQVM mean scores were lower than the PPVT mean scores, but there are some key differences between the 2 measures. The TQVM responses involve linguistic processing of questions and for the PPVT, the child points to a picture that represents the vocabulary item. Still, the correlations in the Kearns and Biemiller study (2010) were substantial with predicted improvement in performance across the 3 grades tested.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

6 Narrative Language/Story Knowledge Measures

Narrative language measures can provide important information on the language proficiency of young children. However, few have been developed for use in multiple languages and cultural contexts. Several measures employ simple approaches that could be adapted for multi-lingual contexts, though. The revised Renfrew Bus Story (Glasgow & Cowley, 1994) and the Test of Narrative Language (Gillam & Pearson, 2004) both include tasks that ask children to listen to stories with oral narration responses. The Test of Narrative Language includes a 5-picture sequence task as well as single picture tasks. The Test of Narrative Language has good technical adequacy. The Renfrew Bus story has limited technical adequacy, however (Hayward, et al., 2008).

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

7 Story retell measures: The Renfrew Bus Story

The Renfrew Bus Story is measure that is administered individually to young children. The purpose of the test is to measure the ability of children to retell events in a story, the Bus Story. The retell provides information on children’s integrative language skills (Renfrew et al., 1994). The Bus Story measures children’s ability to “retell relevant information about a story” (p. 1). Data suggest that it is best utilized as a screening tool since the reliability of the measure is low.

The test kit includes a story booklet, a record form, and the examiner’s manual. A scripted story is read to a student and there are 12 pictures that remain in view of the student while the retell is recorded and questions are answered. The adult can prompt the student for a response (Hayward et al., 2008).

The scoring and interpretation are fairly complex and contribute to the low reliability of the measure. Further, it must be administered individually requiring substantial administrative time, with scoring requiring a transcription of the student’s recorded response. Thus, the test is not feasible to use for impact evaluations.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

8 Picture Story Retell: the MAIN assessment

The Multilingual Assessment Instrument for Narratives or MAIN (Gagarina et al., 2012) is a tool that can be administered to a child in more than one language to assess language skills in each language. The prompts include 4 sets of 6 pictures that comprise each story. They are constructed so that they are folded from picture 1 to picture 6. There are 3 options for administering the task: the assessor can model the story, picture by picture and then ask the child to retell the story, the child can be asked to tell the story, or the assessor can tell the story in pictures and ask the child comprehension questions about the story. The scoring protocol is detailed and requires careful recording of the child’s response. The assessor must be quite skilled in order to score the retell accurately. However, the option of telling the story in pictures and asking comprehension questions is much simpler to score.

The MAIN has been developed in a number of different languages, but, there is no evidence to date on its validity and reliability for use in developing countries. However, it has promise as a tool for assessing the language skills of young children as some of the picture sets seem appropriate for different contexts. For impact evaluations, modeling the story through the pictures and asking comprehension questions is probably the most efficient way to use this assessment tool. It could be an additional task for an EGRA assessment that provides important language comprehension information with picture cues. The comprehension questions are asked with the story pictures visible to the child.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

9 A new task for the EGRA measure in rural Liberia

Because of the need to create a task with simple instructions to maximize accurate administration and scoring, and a task that did not require the assessor to undergo extensive training, or to have a strong background in language development, a decision was made to adapt the TQVM as a subtask for an EGRA assessment in a study in rural Liberia.

A pilot version of the TQVM was developed by Concern Worldwide for a USAID-funded early grade reading study in rural Liberia. The study forms part of the baseline assessment for the Education Quality and Access in Liberia (EQUAL) project implemented by Concern Worldwide. The modification resulted in renaming the “EGRA” as EGLA, Early Grade Literacy Assessment, and was supported by experts from Cambridge Education and SIL International. The new EGLA test was piloted in May 2014 – findings of this study are presented in this paper to document the utility of the additional TQVM sub-test as assessment tool for systematic analysis of receptive vocabulary levels of children in early primary school within developing countries.

Children in Grand Bassa County where the EGLA was administered typically speak the Bassa language and learn English when they come to school. We wanted to briefly sample understanding of simple vocabulary words to determine whether children understood English. We also wanted to find ways of measuring vocabulary comprehension to enhance the data set for a comparative study on literacy interventions in mother-tongue (Bassa) and English being planned under the same project. Finally, by expanding the EGRA tool to EGLA we intended to enhance dialogue and decision-making within the programme and with the Ministry of Education on the needs of students to move from low reading competencies to proficiency in reading through the inclusion of language comprehension.

A receptive vocabulary sub-task was added to the existing EGRA tool (the original EGRA tool was developed by RTI for the Liberian context and modified over the past five years). The new sub-task was developed remotely by experts from Cambridge Education and SIL, then piloted and revised in Liberia with word lists and input from experienced Liberian assessors. The TQVM can be developed as a general vocabulary measure when it taps into key words children should know the meanings of across the primary and intermediate grades. It was not intended to be administered frequently since the words selected are not words that are to be directly taught and then tested. However, variations of this measure can be created to use to check whether children have learned the meanings of words taught. The vocabulary words were generated by local educators and assessors with experience assessing young children in Liberia on EGRA in English. These words were then generated into six pairs of statements (twelve statements in total). The statements are provided below, in the order read to children:

i. A mountain is very small (no)

ii. Goats eat grass (yes)

iii. Many people use a cutlass to wash their hands (no)

iv. A canoe can be used for fishing (yes)

v. Goats can learn to read (no)

vi. We cut milk (no)

vii. A canoe can be used to dig in the garden (no)

viii. People use a cutlass to cut down the bush (yes)

ix. My mother cooks bicycles (no)

x. A mountain can be seen from far away (yes)

xi. My mother cooks plantain (yes)

xii. We cut rice (yes)

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

During the individually administered oral EGRA, assessors introduced this task using an example of two statements and modelling correct ‘yes’ and ‘no’ responses. When the child indicated that they understood the process, the assessor proceeded with the test by reading each sentence to the child (children did not see the written text), and children were required to say if they thought the statement was true (yes) or false (no). Correct responses are required for both sentences (six ‘yes’ and six ‘no’), with a probability of 0.25 of answering two questions correctly by chance. This follows the scoring mechanism applied by Kearns and Biemiller and is the same probability as the four options multiple choice format in the PPVT-III. Scoring calculations were automatically generated for assessors through the TangerineTM platform, minimizing assessor errors.

The study was conducted with 940 children (581 male, 359 female) in 85 schools across the five districts of Grand Bassa County, Liberia. Originally this was designed as a sample-based study of 100 schools applying multi-stage stratified sampling by location and sex. However, fifteen schools closed two months before the end of the school year, and very high absenteeism within other schools (most schools had attendance rates below 50%) resulted in a revised methodology. Weak management of the academic calendar and high absenteeism are common challenges within this impoverished, mostly rural context and the EQUAL programme is designed to address these. Rather than applying a sample-based survey, the EGLA tool was applied to all grade two and three students present on the day of data collection (census of all children in grades two and three that day, but not representative of a wider population). Across the 85 schools assessed, 37% of children in grades two and three were present and assessed for this study. Schools in districts 2, 3 and 4 are located in rural areas; schools in districts 1 and 5 are located in periurban or rural areas.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

Overall reading levels were extremely low in both grades two and three. Children scored well in the letter naming sub-task (identification of letter names) with a mean score of 69.6 correct letters per minute (CLPM) in grade 2 (SD = 20.73) and 84.2 CLPM (SD=22.79). However, students struggled to identify letter sounds in the phonemic awareness sub-task where they were asked to identify words that started with a different sound

mean scores were 43% and 51% correct responses for grades two and three respectively (SD = 10.7 and 17.0 respectively). When presented with unfamiliar (nonsense) words to assess lettersound correlations for coding/de-coding students struggled to read any words. In grade two 94% of students could not read any words in the list, and in grade three 77% of students were unable to read a single word.

For the oral reading fluency sub-task mean scores for grades two and three respectively were just 10.4 correct words per minute (CWPM) and 19.9 CWPM (SD = 10.68 and 17.07 respectively). Mean scores for boys were slightly higher than girls for both grades as provided in the table below. This is more considerable when the ratio of girls to boys tested is considered (0.62) against the gender parity ratio of enrolment for girls and boys in the target schools (0.72). This indicates that fewer girls were enrolled; of those enrolled fewer girls attended school on the day of data collection, and of those attending mean scores for girls were lower than for boys. Scores for boys also fall far below the government interim target for Liberia (45-65 CWPM).

The range in scores is considerable, with a minimum score of zero CWPM in both grades and maximum scores of 61.1CWPM in grade two and 105 CWPM in grade three. One in five students in grade two (20.9%) could not read any words in the passage, and 7.53% of students in grade 3 scored zero in the same sub-task.

Following the oral passage reading fluency sub-task students were asked up to five comprehension questions based on the text. The first four questions asked for information provided directly in the text, the fifth required the student to apply reasoning skills (“why is John happy?”). This could be used to gain some information on student’s language comprehension, but due to the low reading levels of students it is not possible to draw conclusions on their level of comprehension. This is due to the ‘skip’ mechanism built into the EGRA test (which is a valuable mechanism to avoid undue stress for children) whereby if a child does not succeed in reading any part of the passage, he/she is not asked any comprehension questions on the passage. If the child reads to a certain point, he/she is only asked questions relevant to the part of the passage read. In this case 17.4% of students were never asked any comprehension questions and only 4% of students were asked all five questions. Without a significant number of responses we are unable to

Table 1: Student Scores in Oral Reading Fluency (CWPM)

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

draw any conclusions on the language skills of the majority of children tested, irrespective of their reading levels.

Results from the TQVM, or oral vocabulary sub-test as it was labelled in the EGLA are questionable. Student scores are relatively high in this sub-test. From a possible score of six correct word-pairs, the mean score for students in grade two is 4.1 (SD=1.34) and in grade three is 4.7 (SD=1.22). The distribution of scores differs from other sub-tests – in this case students at the first and third quartile demonstrate scores in the upper range (three or more correct responses). Only 2.6% of students in grade two scored zero and 0.46% of students in grade three scored zero. Overall results for this sub-test are provided below.

The high scores of students in this sub-test and lack of range in scores present difficulties. It is contrary to the observations of the assessors and language experts involved – children tested did not demonstrate the language capacities aligned with the test results during observations, nor do they appear to have opportunities to develop the vocabulary in their current educational or home situations. This may be a misperception and needs to be verified through further assessments to ensure that education responses are aligned with the actual levels of students.

If the results are a true reflection of student oral vocabulary levels, the TQVM has not met the purpose for which it was designed in this situation – without a significant span in scores we can conclude very little from the data in terms of baseline values (i.e. scores of the baseline test are very high rendering targets less meaningful), and we cannot derive correlations between scores in this sub-test and other sub-tests of the EGRA to predict reading levels. At best, we are provided with a positive perspective – children have more English vocabulary than expected. At worse, we are provided with flawed results. From our observations of the children in this context this seems to be the most likely scenario, unfortunately. However, further assessments are needed to verify this hypothesis.

However, we believe that this process provides learning for the education community on the utility and future variations of the TQVM within an EGLA, which can benefit future iterations of oral vocabulary assessments for similar contexts. Provided below are two limitations of the test which may be attributable to the lack of utility in the results.

Delivery of the assessment: To align with the Early Grade Reading Assessment the delivery of the TQVM differed from the format original delivery by Kearns and Biemiller. They provided students with response sheets and students marked the happy or frowning rebus face to indicate if they thought the statement was true or false. As the test was administered to the whole class, assessors read the

Table 2: TQVM Oral Vocabulary Correct Responses (max score =6)

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

statements to the group, rather than one-to-one communication. In the EGLA administration students sat individually with the assessor and provided oral responses of yes and no. There is a risk that unintentional cues were provided by assessors through facial cues, tone and gestures which could have supported students in guessing the correct response. Further, there is a risk that intonation or even minor translations of the sentences (in the pronunciation or form of some words) into the non-standard Liberian English widely used in Liberia provided additional cues. This may have resulted in interpretative tonal inflections in the sentence that could indicate if the sentence was true or false.

Word list selection: The original TQVM applied by Kearns and Biemiller used twenty-two words (resulting in forty-four test items) taken from PPVT-III levels considered appropriate for ages four through sixteen years. The TQVM sub-test in the EGLA tool was limited to just six words (twelve test items) to avoid adding considerable time to the EGRA. This is an important consideration – as an oral individually administered test it is important that children are not expected to concentrate for lengthy testing. However, when we examine the words used in the EGLA TQVM and the corresponding results, the list may too limited by length and difficulty level. There is no verified vocabulary list available for this context, such as those tested and verified by Peabody in the United States, which presents challenges in word selection for Liberia. The words selected for the EGLA may be more suitable for reading vocabulary at grade two and three levels than for oral vocabulary. Given the focus on grade-level reading vocabulary in the EGRA it is likely that the Liberian experts developing the word lists applied similar words to this test. The oral vocabulary expected for use in the classroom to explain concepts and express ideas is likely to be more complex, with the inclusion of basal and ceiling words. Validation of the test using an expanded word list (increased number of words and increased range of difficulty), preferably with vocabulary appropriate to the context, could potentially provide a richer data set more reflective of student levels observed.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

11 Recommendations for further TQVM tools

Despite the disappointing data set resulting from this EGLA tool, we experienced certain successes in the process which have value. The introduction of a receptive vocabulary sub-test and the involvement of the EQUAL team, in addition to planning and dialogue with Ministry of Education staff in Liberia, provided opportunities to enrich discussions on literacy with the importance of language comprehension. The simple act of developing and using a tool to assess vocabulary allows for increased focus on oral language development in interventions and programmes. This would have been enhanced with a richer baseline data set from which targets would have been set. The EQUAL programme works in partnership with the Ministry of Education and LIBTRALO, a national-level mother-tongue literacy organization, so the development and use of this tool has provided a new platform on which dialogue and evidence-gathering can build.

Further, the assessment team reported satisfaction with the process of data collection. The inclusion of a short sub-test on receptive vocabulary (embedded into the existing EGRA and rendered onto the digital TangerineTM platform) fits into existing assessment structures without requiring additional sample designs, survey tools or data collectors. Similarly, data analysis is facilitated by the digital transferal of the full EGLA data set from Tangerine – this process has been refined by RTI enabling additional sub-tests to fit into the system with minimal training. While the administration methodology may have been convenient, there is a need to question the reliability of this process – as described above there is a risk of assessor error through cues that needs further analysis.

We conclude that the TQVM may have utility for contexts such as Liberia, to enhance existing reading assessment tools with the inclusion of receptive vocabulary assessment. However, further testing is required to assess the validity of the tool contents and delivery. We recommend the validation of the tool, correlating results against at least one test which accurately predicts student reading levels, such as the PPVT-III, but preferably one developed and validated in a developing country context. During this validation testing, we recommend that researchers refine the delivery mechanism, or apply some form of assessor cue monitoring and control, to verify if the test can indeed be delivered on an individual oral basis. Secondly, we recommend that the word list used include an extended list of vocabulary ranging in difficulty level to capture basal and ceiling words. We would also encourage researchers to consider mechanisms to enhance existing tools in use in developing countries, such as UWEZO or ASER, to include vocabulary and language comprehension assessment.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

Annual Status of Education Report, (2015). www.asercentre.org

Biemiller, A. (2003). Oral comprehension sets the ceiling on reading comprehension. The American Educator, Spring 2003, p. 23.

Biemiller, A., & Slonim, N. (2001). Estimating root word vocabulary growth in normative and advantaged populations: Evidence for a common sequence of vocabulary acquisition. Journal of Educational Psychology, 93 (3), 498-520.

Cain, K., & Oakhill, J. (2003). Reading Comprehension Difficulties. In T. Nunes & P. Bryant (Eds.), Handbook of Children’s Literacy (pp. 313-338). The Netherlands: Kluwer Academic Publishers.

Dunn, L.M., & Dunn, L.M. (1997). Peabody Picture Vocabulary Test, Third Edition. San Antonio, TX: Pearson Education.

Florit, E., & Cain, K. (2011). The simple view of reading: Is it valid for different types of alphabetic orthographies? Educational Psychology Review, 23, 553–576.

Gagarina, N., Klop, D., Kunnari, S.,Tantele, K., Välimaa, T., Balciuniene, I., Bohnacker, U., Walters, J. (2012). MAIN Multilingual Assessment Instrument for Narratives. ZAS Papers in Linguistics, 56.

Gillam, R. B. & Pearson, N. A. (2004). Test of narrative language. Austin, TX: Pro-Ed.

Glasgow, C., & Cowley, J. (1994). Renfrew Bus Story test - North American Edition. Centreville, DE: Centreville School.

Gove, A., Chabbott, C., Dick, A. DeStefano, J., King, S., Mejia, J. and Piper, B. (2015). Early learning assessments: a retrospective. Background Paper for Education for All Global Monitoring Report, UNESCO. Paris: France.

Hart, B., & Risley, T.R. (1995). Meaningful differences in the everyday experience of young American children. Baltimore: Brookes Publishing.

Hart, B., & Risley, T.R. (2003). The early catastrophe: The 30 million word gap by age 3. The American Educator, 27,(1), p. 4-9.

Hayward, D. V., Stewart, G. E., Phillips, L. M., Norris, S. P., & Lovell, M. A. (2008). Test review: The Renfrew Bus Story (RBS). MAIN Multilingual Assessment Instrument for Narratives (2012)

Edmonton, AB: Canadian Centre for Research on Literacy. Retrieved [insert date] from http://www.uofaweb.ualberta.ca/elementaryed/ccrl.cfm

Hogan, T.P., Adlof, S.M., & Alonzo, C.N. (2014). On the importance of listening comprehension. International Journal of Speech-Language Pathology (16), 3, 199-207.

Ireton, H. (1992). The Child Development Inventory Manual. Minneapolis, MN: Behavior Science Systems, Inc.

Expanding EGRA: The Early Grade Literacy Assessment and its contribution to language instruction in Liberia

Ireton H and Glascoe FP. (1995). Assessing children's development using parents' reports. The Child Development Inventory. Clin Pediatrics, 34(5), 248-55.

Kearns, G., & Biemiller, A. (2010). Two-questions vocabulary assessment: Developing a new method for group testing in kindergarten through second grade. Journal of Education, 190, 31-41.

Muralidharan, K. (2013). An evidence-based proposal to achieve universal quality primary education in India. Accessed on July 5, 2015 at http://ideasforindia.in//article.aspx?article_id=222

Nag S., Chiat S., Torgerson C., Snowling M.J. (2014) Literacy, Foundation Learning and Assessment in Developing Countries: Final Report. Education Rigorous Literature Review. Department for International Development.

Stark, C. (2011). Building Human Capacities in Least Developed Countries to Promote Poverty Eradication and Sustainable Development. UNESCO. Accessed on July 10, 2015 at http://portal.unesco.org/en/ev.php-URL_ID=37082&URL_DO=DO_TOPIC&URL_SECTION=201.html

Research Triangle Institute International (2009). The Early Grade Reading Assessment Toolkit. accessed on July 10, 2015 at http://www.ineesite.org/uploads/files/resources/EGRA_Toolkit_Mar09.pdf

Scarborough, H. (2001). Connecting early language and literacy to later reading (dis)abilities: Evidence, theory, and practice. In S. B. Neuman & D. K. Dickinson (Eds.), Handbook of early literacy research (pp. 97– 110). New York: Guilford Press.

Tobia, V., & Bonifacci, P. (2015). The simple view of reading in a transparent orthography: The stronger role of oral comprehension. Reading and Writing. Published online on February 15, 2015. DOI 10.1007/s11145-015-9556-1.

UWEZO. (2013). Are our children learning? Literacy and numeracy across East Africa. Uwezo East Africa at Twaweza , accessed on July 13, 2015 at www.twaweza.org .

Whitehurst, G. J., & Lonigan, C. J. (2001). Emergent literacy: Development from pre-readers. In S. B. Neuman, & D. K. Dickinson,(Eds.), Handbook of early literacy research (pp. 11–29). New York: Guilford Press.