PVAT Overview by MHS Assessments

Product Overview Please note: All data presented in this overview are subject to change.

Overview What it Measures: Vocabulary acquisition (receptive knowledge of the meaning of English words presented verbally and with visual stimuli) of both English speakers and English learners. Age Range: 2 years 6 months to 22 years 11 months Administration Time: 10 to 15 minutes Parallel Forms: Two parallel versions (Form A and Form B) are available. The forms are matched in terms of psychometic properties and are appropriate for repeated administration to measure and track growth and development in vocabulary. Qualification Level: B

Revolutionary Dual Normative Samples with Control for English Exposure: 1. English Speaker Norms: Stratified to match the U.S. Census on gender, age, race/ethnicity, parental education level, and region. 2. English Learner Norms: Stratified to match the U.S. Census on gender, age, parental education level, region, language spoken (Spanish and Spanish Creole, Indo-European Languages, Asian and Pacific Languages, and All Other Languages). Data collection targets were also set to represent various the lengths of lifetime exposure to English.

Format: The Ortiz PVAT is available as downloadable software (Windows only) with online scoring. Fully computerized administration and scoring eliminates the manipulation of a stimulus easel. Administration: The Ortiz PVAT is individually administered. Images illustrating a target word and a set of three distractors are presented on a computer screen. The target word is presented in a pre-recorded audio format and the examinee indicates the meaning of the presented word by selecting the picture that best illustrates the object, action, or concept of the target word. Ecologically Valid Visual Stimuli and Neutral Audio Recording: Words representing various parts of speech and levels of proficiency depicted by ecologically valid photos of real objects/actions Figure 1. Sample Ortiz PVAT Item (see Figure 1), without anachronistic or regional features, are presented with clear and neutral pre-recorded pronunciation and voicing.

info.mhs.com/OrtizPVATinfo

Built-in Basal: Testing begins with a Screener that is used to determine the appropriate starting point for each examinee. There is no need for the examiner to determine the basal as this is accomplished automatically within the program. Examiners may, however, choose to skip the Screener and begin the administration at the first item, if so desired. Built-in Ceiling: The computerized program automatically stops when the examinee has reached their ceiling. Parts of Speech and Word Types Assessed: Different parts of speech (i.e., nouns, verbs, adverbs, adjectives, and prepositions) are evaluated, as well as vocabulary associated with Basic Interpersonal Communicative Skills (BICS) and Cognitive Academic Language Proficiency (CALP; Cummins, 1984). Computerized Reports: • Assessment Reports include scores (raw scores, standard scores, confidence intervals, percentiles, age equivalents, and stanines) and interpretive information, as well as instructional level and intervention recommendations. • Progress Reports include a Growth Index for evaluating growth in vocabulary.

Dual Normative Samples The Ortiz PVAT includes both English Speaker and English Learner normative samples that are representative of the U.S. population.

English Speaker Normative Sample

The Ortiz PVAT English Speaker normative sample (N = 1,530) was collected using a stratified sampling plan based on age, gender, race/ethnicity, parental education level, and geographic region. English Speakers, based on the 2014 American Community Survey of the U.S. Census Bureau, were defined as those who indicated “Speak only English” for their language spoken at home. The sample was balanced across 17 age bands (N = 90/age band), and gender was balanced (50% male; 50% female) within each age group. Representation by race/ethnicity, parental education level, and geographic region all fell within 2% of the U.S. population targets (see Tables 1 to 3).

info.mhs.com/OrtizPVATinfo

English Learner Normative Sample The Ortiz PVAT English Learner normative sample (N = 1,190) was collected using a stratified sampling plan based on age, gender, language spoken at home, parental education level, and geographic region. This stratification plan was based on characteristics of individuals in the U.S. population with exposure to a language other than English as reported in the 2014 American Community Survey of the U.S. Census Bureau. The sample was balanced across 17 age bands (N = 70/age band), with a balanced gender split (49.2% male; 50.8% female). Representation by language spoken fell within 2% of the U.S. population targets (see Table 4). In addition, efforts were taken to ensure a wide range of exposure to English. Although no Census data exists about the length of exposure time to English, a balanced proportion of each exposure level was collected to ensure representativeness. Table 5 presents the English Learner normative sample’s exposure to English in terms of absolute duration (i.e., number of months/years with exposure to English), and Table 6 presents the sample’s exposure to English in terms of relative duration (i.e., percentage of life exposed to English, divided into five levels). Given the specificity of the English Learner normative sample, the language spoken by the individual was prioritized in the stratification and correspondence with Census, followed by parental education level and U.S. region (which were matched to the Census figures where possible; see Tables 7 and 8).

info.mhs.com/OrtizPVATinfo

...continued from the previous page

The Importance of Dual Norming and English Language Exposure in the Assessment of English Learners: Explained In addition to being a highly sensitive measure of receptive vocabulary in English, the Ortiz PVAT represents a pioneering way of assessing English learners by using dual normative samples and incorporating differential exposure to English in the English Learner sample. The creation of a separate and distinct normative sample for English learners is not only unique, but is necessary to address questions regarding “difference versus disorder;” doing so permits the comparison of an examinee’s performance to a “true peer” group, which cannot be done by using native English Speaker norms. However, it is not enough to merely use same-age English learners to create an appropriate normative sample, because English learners do not all possess the same level of proficiency and development in English. In other words, some English learners may have considerable exposure to English, whereas others may have very little. For monolingual, native English speakers, the use of age is sufficient to control for such exposure and permits valid comparisons of performance, because development has only occurred in a single language (i.e., English). Thus, all 10-year-old, monolingual, native English speakers have 10 years worth of experience with and exposure to English. This is not the case, however, for English learners who, by definition, have been exposed to two or more different languages at varying degrees. Although any two 10-year-old English learners have had the same total amount of language exposure, the proportion of that exposure to English may vary considerably. A 10-year-old English learner could have as little as three months experience with English, whereas another English learner may have already had 5 years of exposure; since both English learners have very different exposure times to the English language, their vocabulary in English would also be very different. Therefore, because age alone does not control or account for such differences in exposure, and because of the rapid changes that occur during the initial phases of language acquisition, it would be inappropriate and inaccurate to group all English learners of the same age together and expect comparable performance. In light of this important fact, it was necessary to create a normative sample for English learners that directly accounts for, and controls for the level of exposure to and experience with English. The creation of this sample was essential for measuring vocabulary acquisition and growth in a valid way, and for examining the “difference versus disorder” question that is fundamental to all evaluations with English learners. Comparisons to other English learners with similar exposure time to English can only be performed in this manner in order to ensure that each comparison is truly fair and nondiscriminatory with respect to the rate of vocabulary acquisition (which is strongly related to general English language development and proficiency). ...continued on the next page Copyright © 2018 Multi-Health Systems Inc. All rights reserved.

info.mhs.com/OrtizPVATinfo

...continued from the previous page

In order to properly model the rate of vocabulary acquisition and varying ability levels associated with different levels of exposure to English, it was critical to account for exposure to English in the scoring procedure for English Learner normative scores. When scoring English Learners using the same algorithm to compute English Speaker normative scores, the need for separate norms became clear. As seen in Figure 2, when compared against the English Speakers, individuals in the English Learner sample scored consistently below 100 (the English Speaker normative sample mean), especially when exposure to English was low (i.e., 0–10% of their lifespan). However, when scored with the English Learner norms, English Learners in the normative sample obtained scores within the Average range (standard score = 90–109). The group with the highest exposure to English (i.e., 50–100% of their lifespan) most closely mirrors scores from monolingual English Speakers, as expected due to their substantial amount of exposure to English. These results demonstrate that English Learners achieve developmentally typical scores when compared against a valid reference group of multilingual peers when exposure to English is factored into the scoring process. This finding supports the validity of dual norms and the necessity of creating two reference samples, emphasizing the importance of a fair comparison to a relative group when interpreting scores.

info.mhs.com/OrtizPVATinfo

Reliability Alternate Form Reliability

The alternate versions of the Ortiz PVAT (Form A and Form B) were carefully developed to be parallel in terms of both content and psychometric properties. To test the parallel nature of Form A and Form B, alternate form reliability coefficients, as measured by Pearson’s correlation coefficient (corrected for range instability), were calculated. Results provided evidence for excellent alternate form reliability for both the English Speaker and English Learner normative samples (corrected r = .991– .996, p < .001). This finding supports the parallel nature of the two forms, such that scores from either form can be considered reliable and be used interchangeably, or as a tool for tracking progress over time.

Internal Consistency

The marginal reliability coefficient of the Ortiz PVAT is .98 (for both Form A and Form B) across the normative samples and .99 (for both Form A and Form B) in a clinical sample. Marginal reliability values above .80 indicate sufficient reliability; therefore, the results from the Ortiz PVAT provide evidence of excellent internal consistency.

Test-Retest Reliability

Test-retest reliability was assessed by obtaining the correlation between Ortiz PVAT standard scores from individuals in the English Speaker and English Learner general population samples who were administered the assessment twice within a 2- to 4-week interval. Results suggest that the Ortiz PVAT has very strong test-retest reliability in both samples (corrected r = .72–.81, p < .001).

Validity Content-Related Validity

Content-related validity of the Ortiz PVAT was assessed through extensive subject-matter expert review. The panel of experts included speech-language professors and school psychologists who had extensive knowledge in language development and the assessment of culturally and linguistically diverse populations. These experts independently inspected the test content for appropriateness with respect to the construct of receptive vocabulary, including item difficulty for the intended age range, visual depiction of the stimuli and distractors, and potential bias related to gender, race/ethnicity, cultural background, or socioeconomic status. Test content was reviewed to ensure that the items appropriately sample the construct of interest and are suitable for the test’s intended use.

Internal Structure

The structure of the items and underlying model of the assessment was investigated as an assumption of the Item Response Theory (IRT) analysis and to provide evidence for the internal structure of the Ortiz PVAT (AERA, APA, & NCME, 2014). The Ortiz PVAT was hypothesized to assess a unidimensional construct, best explained by a single-factor model, as all items are designed to measure the same central construct of receptive vocabulary skill. The dimensionality of the Ortiz PVAT was explored using full-information item factor analysis (analyzed via the mirt R package; Chalmers, 2012). Results supported a one-factor model with high factor loadings for all items (values ranged from .428 to .956).

Relation to Clinical Diagnoses

In order to extend the results beyond a general population, a clinical validation study was conducted to explore the appropriateness of using the Ortiz PVAT with special populations. The sample included the following clinical groups that were hypothesized to have difficulty with receptive vocabulary acquisition and, therefore, lower scores on the Ortiz PVAT: • Language Disorder with Receptive impairment or or Mixed Receptive-Expressive Language Disorder (n = 50), and • Intellectual Disability (n = 31). ...continued on the next page Copyright © 2018 Multi-Health Systems Inc. All rights reserved.

info.mhs.com/OrtizPVATinfo

...continued from the previous page

Additionally, data were collected from the following clinical groups that were not expected to have problems, at a group level, with receptive vocabulary ability and were, therefore, not expected to have lower than average scores on the Ortiz PVAT: • Language Disorder with Expressive Impairment (n = 52), and • Attention-Deficit/Hyperactivity Disorder (ADHD; n = 31). Participants in the clinical sample were categorized as monolingual English Speakers. A randomly selected sub-sample of the English Speaker normative sample (General Population; n = 52) was included as a reference sample and comparison point for scores from individuals without language-related impairment. Figure 3 summarizes the results of the analyses of covariance that were conducted to compare Form A standard scores between the clinical and General Population groups (nearly identical results were found for Form B). Overall, results confirmed the expectation that certain groups (i.e., Intellectual Disability and Langauge Disorder [Receptive or Mixed]) would have scores that showed difficulties with receptive vocabulary acquisition (as demonstrated by lower scores on the Ortiz PVAT), while other groups (i.e., Language Disorder [Expressive] and ADHD) had average Ortiz PVAT scores with no clear indication of receptive vocabulary difficulties.

References American Educational Research Association, American Psychological Associate, & National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. Washington, D.C.: AERA. Chalmers, R., P. (2012). mirt: A Multidimensional Item Response Theory package for the R environment [R. package version 1.17.1]. Journal of Statistical Software, 48(6), 1–29. Cummins, J. (1984). Bilingualism and special education: Issues in assessment and pedagogy. San Diego, CA: College-Hill.

info.mhs.com/OrtizPVATinfo