AdultSkillsAssessement

For Official Use

DEELSA/ELSA/ED/CERI/CD(2002)3

Organisation de Coopération et de Développement Economiques Organisation for Economic Co-operation and Development

14-Jun-2002 ___________________________________________________________________________________________ English - Or. English

DIRECTORATE FOR EDUCATION, EMPLOYMENT, LABOUR AND SOCIAL AFFAIRS

DEELSA/ELSA/ED/CERI/CD(2002)3 For Official Use

EMPLOYMENT, LABOUR AND SOCIAL AFFAIRS COMMITTEE EDUCATION COMMITTEE CERI GOVERNING BOARD

ADULT SKILLS ASSESSMENT

English - Or. English

JT00128327

Document complet disponible sur OLIS dans son format d'origine Complete document available on OLIS in its original format

DEELSA/ELSA/ED/CERI/CD(2002)3

TABLE OF CONTENTS

ADULT SKILLS ASSESSMENT ..................................................................................................................3 Introduction .................................................................................................................................................3 What skills should be assessed? ..................................................................................................................5 What adult population should be covered?..................................................................................................6 Management and links between PISA and Adult Skills assessments..........................................................6 Timelines .....................................................................................................................................................7 Costs ............................................................................................................................................................8 ANNEX RECOMMENDATIONS OF THE STRATEGIC MANAGEMENT GROUP REGARDING ADULT SKILLS MEETING OF 31 JANUARY – 1 FEBRUARY 2002....................................................10

2

DEELSA/ELSA/ED/CERI/CD(2002)3

ADULT SKILLS ASSESSMENT

Introduction 1.

The International Adult Literacy Survey (IALS) , implemented between 1994 and 1998 in 18 OECD countries and 2 non-OECD countries, was the first internationally comparable, direct assessment of skills of the adult population of working age, in particular literacy skills. It demonstrated, on the one hand, that educational qualifications were an imperfect measure of cognitive skills, but also that labour market rewards covered a broader panoply of skills than those conveyed by literacy, as defined and measured in IALS. This pioneering effort suggested that direct measurement of adult skills by means of home-based assessments was both feasible and yielded information that was useful for analysis and policy.

2.

IALS was conducted by the OECD with leadership provided by Statistics Canada and the US National Center for Education Statistics (NCES). It assessed: − prose literacy; − document literacy; and − quantitative literacy.

3.

A follow-up of IALS, called the International Life Skills Survey (ILSS), was then developed by Statistics Canada and NCES and, in 1999, taken forward in collaboration with the OECD Secretariat as an activity in which countries were invited to join. At that stage, Statistics Canada and NCES had commissioned international teams to develop assessment materials in: prose literacy (using the IALS framework); document literacy (using the IALS framework); numeracy (replacing the IALS quantitative literacy); problem solving; information and communication technology (ICT) literacy; teamwork skills; and practical cognition or tacit knowledge (based on the research of Robert Sternberg).

4.

Despite the investment of considerable expertise and resources, pilot work revealed that adequate measures in all of these domains could not be developed within the timeframe required for inclusion in the proposed international survey. The practical cognition measure was dropped, the proposed direct measures of teamwork and ICT literacy were replaced by components in the background questionnaire through which respondents would be asked to report on their involvement with teams and their use of ICT, and the scope of the problem-solving domain was reduced to analytical reasoning. To signal the reorientation, the name of the survey was changed to the Adult Literacy and Lifeskills (ALL) survey.

3

DEELSA/ELSA/ED/CERI/CD(2002)3 5.

Both OECD and non-OECD countries have been involved in the development of the instruments for ALL. Pilot work and field trials have been undertaken and countries will be collecting full survey data in the period from July 2002 to June 2003. For those participating countries that had earlier participated in IALS, the ALL survey will provide a second assessment of adult literacy levels, while for the rest it will provide an initial assessment. In view of the conversion of the quantitative scale used in IALS to a welldefined numeracy scale in ALLS as well as the introduction of an assessment of problem-solving, ALLS will serve as a invaluable bridging link between IALS and an eventual OECD-coordinated adult assessment. The possibility of there being a second round of the ALL survey after 2003, to enable additional countries to participate, has been suggested in the DeSeCo strategy paper.

6.

In December 1997, with the second round of IALS completed and planning underway for the Programme for International Student Assessment (PISA) that would survey 15-year-olds at school in reading, mathematics and science literacy, a new project was initiated under the OECD Indicators of Education Systems (INES) to step back from the immediate engagement in the development of assessments to develop a conceptual framework for work on competencies. The Definition and Selection of Competencies (DeSeCo) project was led by the Swiss Federal Statistics Office and the US NCES. 7.

The INES General Assembly in Tokyo in September 2000 and the OECD Education Ministerial meeting in April 2001 invited the Secretariat to examine possibilities for the further assessment of adult skills. Any further survey will clearly need to build on the conceptual work of DeSeCo and the empirical work of IALS and ALL. 8.

Any new adult skills assessment programme will require a significant lead time from the moment it is decided to commence exploratory developmental work, particularly if new domains of skills are to be added. PISA took five years from initial commitment to data collection with assessments in well-defined domains and a school-based data collection. The ALL survey has taken five years from commitment to data gathering with instruments that extend beyond those of IALS in limited ways. Another survey on adult competencies is likely to require more than five years between commitment to begin exploratory development work and eventual data gathering. It is clear, therefore, that some broad discussion of possible options needs to begin if the Ministerial mandate for adult skill assessment is to proceed in a way that would permit timely introduction of an expanded adult assessment survey, if this is what countries opt for.

9.

A preliminary scoping paper on the question of adult skill assessment was prepared for the INES Strategic Management Group in January 2002, which gave its views on future directions and recommended the convening of this meeting (see Annex 1). The purpose of this meeting is to obtain national input for a more substantial overview document on options for the future, to be circulated to countries for discussion and presentation to the Education Committee, the CERI Governing Board and the Employment, Labour and Social Affairs Committee in October-November 2002.

10.

Shortly after the January 2002 SMG meeting, the Second International DeSeCo Symposium was held. One of its main objectives was to outline options and strategies that contribute to the eventual development of a coherent assessment programme for key competencies of young people and adults in an international context. Underlying this effort was an interest in broadening assessment to include a wider range of competencies. The conclusions of the DeSeCo project will be presented in a strategy paper to the Education Committee, the CERI Governing Board and the Employment, Labour and Social Affairs Committee in October-November 2002. A draft of that paper has has been made available for this meeting.

11.

In order to help the Secretariat lay down more clearly what form an eventual assessment might take and to formulate a proposal accordingly, there are a number of concrete issues on which national views are required. This will facilitate the specification of a proposal that is informed by both national requirements and constraints. Whether that proposal goes forward is a question for the future. The aim of

4

DEELSA/ELSA/ED/CERI/CD(2002)3 this meeting is to define in broad terms its main parameters. The views expressed at this meeting do not commit countries to a future survey of adult skills. That will be a matter for the relevant committees of the OECD, if indeed preliminary developmental work is authorised in October-November 2002. The purpose of this meeting is to provide country input into the document to be considered by the Committees in October-November 2002.

What skills should be assessed?

12.

The key question underlying any eventual assessment of adult skills is “What skills should be assessed?”. Up to now, adult assessments have focused largely on measures of literacy, a competence that falls under the general rubric of “cognitive skills”. As central as these skills would appear to be, IALS results have shown that they explain only part of the variation in earnings of adults. Other cognitive skills (e.g. planning, organisation) or other skills embodied in formal qualifications (e.g. self-discipline, punctuality) may play a significant role. DeSeCo concludes that assessment needs to focus on “constellations of competencies” rather than single competencies and, in particular, that more importance needs to be given to the competencies of “acting autonomously” and “joining groups”. The SMG, in its comments, noted the significance of non-cognitive skills including teamwork and communication skills.

13.

The problem this raises for a future adult assessment is that while there is considerable experience, both national and international, in the assessment of cognitive skills1, this is much less the case with non-cognitive skills such as teamwork or communication skills. The recent experience in the ALL survey has shown how expensive, time-consuming and difficult it is to move beyond cognitive skills (such as literacy, numeracy and problem-solving) to measures of skills such as teamwork. Assessments of “interpersonal understandings” have been developed by the Australian Council for Educational Research for its Graduate Skills Assessment but it is unknown to what extent these would carry over internationally.

There are some well-known questionnaire-based psychological instruments for assessing traits such as achievement motivation or authoritarian attitudes, but here again, whether such approaches carry over into international settings to yield comparative measures is uncertain.

14.

This summarises succinctly the current state of affairs with respect to adult skills, its scope and the potential for future assessment. The cognitive skills measured thus far appear central but have limited explanatory power with respect to earnings. Non-cognitive skills represent a largely uncharted and challenging area, for which there is demand as well as (indirect) empirical evidence of their importance.

Questions for discussion: What should be the balance between cognitive and non-cognitive skills in any eventual assessment?

For what cognitive and non-cognitive skills should assessment be attempted?

Should framework and instrument development for new skills, in particular non-cognitive skills, be an integral part of an adult skills survey plan or should this be pursued on a parallel or an upstream research track?

1

This is not to deny that there remain problems with accurately measuring the levels of literacy of persons at the low end of the scale or of persons whose native language is not the language of assessment. The complementarity or compensatory nature of different types of skills may be a critical issue for such persons.

5

DEELSA/ELSA/ED/CERI/CD(2002)3 Should implementation of any further adult assessment be conditional on the possibility of assessing at least one key non-cognitive skill?

What adult population should be covered?

15.

Both IALS and the ALL survey cover the working-age population (persons 15-64). Any further adult skills assessment could continue this practice or could focus on a narrower age range. Certain types of skills (e.g. ICT) may be difficult to test for certain persons at the upper age of the 15-64 age range, except in fairly rudimentary forms (e.g. use of Automatic Teller Machines). Moreover, the possibility of policy or even private intervention on certain types of skills may be limited for adults. Simpler traditional measures such as educational attainment may be sufficient to identify populations at risk. A focus on a narrower age band will also have favourable cost implications, among other reasons by reducing sample size requirements. More could then be invested on broadening the scope of assessment .

16.

On the other hand, the identification of labour market skill requirements through analyses of what employers reward may have implications for adult training systems and, further upstream, for what education systems emphasise and seek to develop in their students. Even if the scope for policy intervention regarding adult skills may not always be obvious, assessments of them provide a skill profile of the general population that is more current than qualifications generally obtained many years prior in early adulthood. They provide the opportunity to determine whether national lifelong learning strategies are indeed effective. In addition, with aging work forces there is growing interest in the skills of older workers and in how much training they invest in. This would also argue for maintaining the broad working-age population focus.

Questions for discussion: What age range should should an adult assessment cover? Given the expense of an adult skills assessment, might there not be a possible trade-off between breadth and coverage, that is, narrowing the target age range and increasing the domains to be assessed?

If the target population were to be narrowed, what would be the appropriate age group to consider?

Management and links between PISA and Adult Skills assessments 17.

Many of the skills planned for assessment in PISA might also be considered / implemented for adult assessment. For example, the ALL survey assessments include an indirect measurement of information and communication technology skills. The PISA 2006 assessment will incorporate a computerbased assessment of ICT skills, if its feasibility can be established in time.

18.

The PISA perspective on competencies embodies the view that education prepares young people for adult life and that the skills assessed for 15 year olds are, thus, not fundamentally different from those that would be assessed for adults. If this is the case, then there is an argument to be made for establishing links between youth and adult assessment. The nature of these links would need to be specified.

Government stakeholders for adult assessment (labour and industry as well as education ministries) are more numerous than for assessment of 15-year-olds; thus representation in decision-making bodies would need to be broadened.

6

DEELSA/ELSA/ED/CERI/CD(2002)3 19.

Some rationalisation of activities might be envisaged. For example, one option might be a pooling of resources for framework and instrument development in areas of common interest. In practice, this might mean separating framework and instrument development from survey implementation.

Questions for discussion: How and to what extent is it possible in designing adult assessments to build upon previous work carried out for students? Are the skills assessed for youth and for adults sufficiently similar to justify the common development of assessment frameworks and instruments?

What form, if any, should an integrated assessment strategy take? For example, should framework and instrument development be placed on a separate track from that of survey implementation, to allow for a pooling of resources in these areas?

What form should a management / decision-making body take? How can representation from the diversity of ministerial stakeholders be ensured?

Timelines 20.

Developmental work on what has become the ALL survey began in 1998, with data collection to take place in late 2002 and early 2003. Over this period, two new direct assessment domains were developed (numeracy and problem solving) and two indirect assessment domains (teamwork, ICT literacy). Based on this experience, a minimum period of five years, roughly speaking, would seem to be necessary between project launch and data collection in a situation where a certain number of new assessment instruments need to be developed.

21.

Should there be agreement to go ahead with a new OECD adult skills assessment, the earliest feasible launch date for development work would appear to be towards mid-2003. The actual data collection date would depend on the timeline for the development of frameworks and instruments for new assessment domains but, in view of the experience of ALLS, would seem unlikely before 2008. Actual decision points for commitment to development work and implementation will need to be identified, based on the agreed-upon project launch date, if any.

22.

For European Union countries, the Lisbon summit has set goals regarding human resources development for the year 2010. A round of skill measures for the adult population would be an important contribution to the 2010 assessment of progress since Lisbon, particularly for countries having participated in IALS or the ALL survey.2 23.

It is expected that an ICT skill assessment will be ready for the 2006 PISA assessment, provided computer assessment is feasible. There are also preliminary discussions to include assessments of communication and interpersonal skills in the 2009 PISA assessment. Although assessment frameworks and instruments do not yet exist for these domains, the PISA timeline for these is indicative of what is being envisaged for a continuing survey that is in the field and with an R&D infrastructure in place.

2

At a recent meeting of European Directors of Social Statistics a project to be launched in 2004 on “Defining A Strategy on Direct Skills Assessment” was approved, where it is specified that “this work … should involve OECD.”.

7

DEELSA/ELSA/ED/CERI/CD(2002)3 Questions for discussion: What implementation calendar would you envisage, keeping in mind: a) the ALL survey data collection date of 2002 / 2003; b) the possible data collection date of no earlier than 2008 for a new OECD assessment launched in 2003; c) the time required to develop assessment frameworks / instruments for new (non-cognitive) skills. d) the possible periodicity of an eventual adult assessment; e) the desirability (or not) of having major / minor domains of assessment (as in PISA), under the assumption of periodic assessments.

Costs 24.

Because of the household-based assessment methodology, in-country costs for adult assessments are higher than for school-based assessments such as PISA. Per respondent costs for IALS ranged from as low as twenty-five dollars to several hundred dollars, depending on national wage rates and whether or not the national statistical agency could be enlisted in the survey implementation work. However, this assumes a household-based assessment. If assessment is to involve such competencies as teamwork abilities, it is no longer clear that the household (rather than, say, a community setting) is the appropriate setting for assessment.

25.

Required sample sizes increase with the number of domains to be assessed, with some variation depending on the performance of the instruments. Sample sizes for IALS with its three domains were in the vicinity of 3000 persons. Sample size requirements for ALLS with its four domains have been set at 5400 persons. By comparison with interviewing and testing costs, international overheads appear relatively inexpensive. Still, the latter are not insignificant and the major factors affecting costs need to be considered.

26.

There is an upper limit of about one and one-half hours of assessment time; beyond this, response rates deteriorate. With this amount of assessment time, about five or six domains can be accommodated.

Each of these will have significant development costs associated with it, unless existing instruments can be used or adapted. Such costs are likely to be even higher for frameworks and instruments for assessing noncognitive skills.

27.

A second point concerns the cost of quality control of task equivalence and of survey operations in participating countries. Although much of the international overhead costs involve the development of new assessment frameworks and instruments, the international costs of ensuring and verifying quality are not insignificant.

28.

The experience with PISA suggests that realistic international overhead costs over 4 years, including the full cost of framework and instrument development for two new domains plus the use / adaptation of existing instruments in three other domains, would be in the vicinity of about 15 to 20M euros, spread over about six or seven years. This would cover as well the costs of extensive quality control procedures, of project management by the Secretariat and of publication and analysis of the results. The country share can be estimated using the PISA allocation percentages. The actual share would of course

8

DEELSA/ELSA/ED/CERI/CD(2002)3 depend on the extent of country participation. Costs would be reduced if there were fewer assessment domains overall and fewer new domains. Full framework and instrument development costs for an entirely new domain are estimated to be about 4M euros.

9

DEELSA/ELSA/ED/CERI/CD(2002)3 ANNEX RECOMMENDATIONS OF THE STRATEGIC MANAGEMENT GROUP REGARDING ADULT SKILLS MEETING OF 31 JANUARY – 1 FEBRUARY 2002

The Strategic Management Group NOTED: •

the potential importance of the DeSeCo symposium to the issue;

•

the need to engage key actors across government in the development of new proposals, including education and labour ministries, the social partners, and, possibly industry ministries;

•

the significance of ‘soft’ and non-cognitive skills including teamwork and communication skills;

•

the great policy interest in ICT skills;

•

that skills requirements change over time and vary across countries in complex ways determined by both culture and technology;

•

that measurement of skills was of most value when it could be linked to clear policy interventions;

•

the need, in view of the cost and complexity of the issue to proceed with caution;

•

the strong reservations by one member of SMG, who felt that further OECD development work, mooting the option of a further survey some years hence, might threaten the willingness of countries to participate in the current ALL survey;

•

that, while recognising that thinking was at a very preliminary stage, the majority of SMG members were attracted to the idea of undertaking a new survey, while believing that it should not be too ambitious in its scope;

AGREED that the current paper should be made available to the Joint Session of the CERI Governing Board and the Education Committee and the meeting of the Employment, Labour and Social Affairs Committee as a basis for an initial discussion of the issue of assessment of adult skills; and RECOMMENDED to the Joint Session of CERI Governing Board and the Education Committee and the Employment, Labour and Social Affairs Committee, that a further paper, based on that presented to SMG, should be prepared for Autumn 2002. This new document should: •

taking account of the conclusions of the DeSeCo symposium, propose a strategy for defining the skills which are to be assessed in the light of feasibility, cost and policy relevance;

•

further clarify the different strategic options and their associated costs;

•

propose mechanisms for ensuring continuing country involvement, with wide representation across government, in the development of proposals for the assessment of adult skills, including a mechanism 10

DEELSA/ELSA/ED/CERI/CD(2002)3 for establishing an appropriate Board of Participating Countries in the event that a sufficient number of countries wish to proceed.

That the Secretariat convene a meeting of national representatives in early summer to work on the development of this new discussion document.

11