/SEML2_wk2_Salmi1

Benchmarking Tertiary Education Systems: a Proposed World Bank Approach Jamil Salmi 14 June 2010

obsession with rankings

my university is‌ more world-class than yours

outline of the presentation • advantages and limitations of institutional rankings • rankings of university systems: a methodological critique • benchmarking: a conceptual framework • next steps and challenges

early experiences with rankings • US News and World Report (1983) • Shanghai Jiao Tong University Academic Ranking of World Universities (2003) • The Times Higher Education Supplement-QS World University Rankings (2004) • Webometrics (2004) • The Higher Education Evaluation and Accreditation Council of Taiwan’s Performance Ranking of Scientific Papers for World Universities (2008)

ranking systems in 2009 Region Eastern Europe and Central Asia East Asia and Pacific Latin America and the Caribbean

National and International Ranking System Kazakhstan (A, B), Poland (C), Slovakia (B), Romania (B/C), Russia (B, IB), Ukraine (B/C) Australia (B), China (B, C, IB), Hong Kong (C), Japan (B, C), Korea (A), Malaysia (A), New Zealand (A), Taiwan (B), Thailand (A) Argentina (D), Brazil (A), Chile (C,D), Mexico (B), Peru (B)

Tunisia (A)

Middle East and North Africa North America

Canada (B, C, B/C), United States (C, IC)

South Asia

India (C/D), Pakistan (A)

Sub-Saharan Africa

Nigeria (A)

Western Europe

France (IB), Germany (B/C, C), Italy (C), Netherlands (A), Portugal (C), Spain (B, C, IC), Sweden (C), Switzerland (B/C), United Kingdom (A, B, IC)

who prepares the rankings?

6 Gouvernement 5

Organisme indépendant Média

4 3 2 1 0 Europe de l'Est et Asie centrale

Asie et Pacifique

Amérique latine et Caraïbe

Afrique et Moyen-Orient

Europe de l'Ouest

Amérique du Nord

a false precision “Could there not…be a move…away from the false precision?

Could you not do away with rank ordering and overall scores, thus admitting that the method is not nearly that precise and that the difference between #1 and #2 - indeed, between #1 and #10 - may be statistically insignificant?”

Stanford University President to Editor of U.S. News (1996)

Anglo-Saxon bias

9

methodological problems • validity of criteria • research bias, hard sciences, English language publications, peer evaluations

• validity of weights given to various criteria • statistical validity • weights and aggregate ranks

methodological problems (cont.)

• do all the indicators taken together actually measure quality?

• do the score differences reflect actual quality differences?

• are the rankings comparing the same types of institutions and programs?

danger of rankings • changes guided by rankings criteria – priority given to top students (equity concern) – resource allocation (research)

• mergers (size) • fraud in data presentation or survey participation 12

positive aspects at institutional level  collecting and publishing more reliable data  analyzing key factors explaining ranking  proposing concrete targets to guide strategic planning  entering into mutually advantageous partnerships  seeking to improve teaching, learning and research

country level

government responses • let us make a new ranking (Russia, Ecole des Mines, France / EU) • let us encourage mergers • let us give additional money • concentrate or spread in an equal manner?

• select or make institutions compete?

‘...Australia cannot afford to spread its relatively small resources too thinly. It must invest in niche areas. This means that some universities and some fields should get preferential treatment. If Australia does not have some universities playing at the high end, Australia will fall behind.’ (Gallagher, 2008)

outline of the presentation • advantages and limitations of institutional rankings • rankings of university systems: a methodological critique

QS-SAFE National System Strength Rankings: Evaluating systems on: • System (total institutional performance in ranking) • Access • Flagship • Economics / Funding

2008 QS SAFE National System Strength Rankings Rank

Country

System Access Flagship Economic Overall Score Rank Score Rank Score Rank Score Rank Score

1 2 3 4 5 6 7 8 9 10

United States United Kingdom Australia Germany Canada Japan France Netherlands Korea, South Sweden

100 98 92 95 92 91 87 91 71 75

1 2 5 3 4 6 8 7 16 13

100 94 97 87 93 80 89 83 81 82

1 4 2 7 5 13 6 9 12 11

100 100 99 95 98 98 98 95 96 94

1 2 3 15 5 4 8 14 12 16

100 98 88 93 86 89 82 76 71 64

1 3 7 5 9 6 11 13 16 19

100 98 94 92 92 90 89 86 79 79

assessment: QS-SAFE • pioneer in looking at country systems • but… all the defects of the THES ranking • reputation a key data driver • methodological slippage year to year • focus on research-driven institutions, minimizes system diversity (flagship, economics based on spending for WCUs)

21

Lisbon Council University Systems Ranking: Citizens and Society in the Age of Knowledge.

Evaluated 17 OECD countries on 6 criteria: •Inclusiveness •Access •Effectiveness •Attractiveness •Age-Range •Responsiveness

Lisbon Council Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Country Australia UK Denmark Finland USA Sweden Ireland Portugal Italy France Poland Hungary Netherlands Switzerland Germany Austria Spain

Score 30.6 31.1 39.1 40.8 49 49.2 49.2 54.3 60.9 62.2 64.4 64.5 69.6 70.3 72.5 76.4 79.4

assessment: Lisbon Council • pioneer in giving a multi-dimensional ranking • but … face same methodological questions as university rankings – focus on universities – research bias – composite index with arbitrary weights – mixing output and input measures – choice of indicators

25

outline of the presentation • advantages and limitations of institutional rankings • rankings of university systems: a methodological critique • benchmarking: the conceptual framework

cross-country comparisons help increase knowledge

multi-dimension comparisons enriches the diagnosis

top 50 universities ARWU ranking-Top 50: 2009

THE ranking-Top 50: 2009

Western Europe (5) Canada (2)

Australia (6)

Japan (2) Other Asia (5) China (1)

U.K. (5) U.S.A. (37)

Western Europe (6) Canada (3) Japan (3)

U.S.A. (18)

U.K. (8)

adjusting for size

SJTU

social mobility and inequality

High 100

Norway

90 Denmark

Social Mobility

80

Finland

Canada

Sweden

70

Germany

60 50 40 30 20

UK USA

10 Low

0 0

10

20

30

40

50

Low

60

70

80

90

100 High

Income Inequality

reality check • well-performing countries without worldclass universities • time dimension?

– emerging economies vs. mature economies

comparing country rankings WEF

WB K4D

SJTU

USA

Denmark

US (1)

Switzerland

Sweden

UK (4)

Denmark

Finland

Japan (19)

Sweden

Netherlands

Switzerland (24)

Singapore

Norway

Canada (24)

Finland

Canada

France (42)

Germany

Switzerland

Denmark (45)

Netherlands

UK

Netherlands (47)

Japan

USA

Sweden (51)

Canada

Australia

Germany (55)

purpose • need to understand the determinants of performance – no consensus on what countries should do to improve their performance – wide variations in system performance with similar funding levels and common country characteristics

35

comparing Brazil and Chile 40% 38% 35%

Chile

Enrollment rate

30% 25%

24%

Brazil

20% 15% 10% 5% 0% 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

public spending as % of GDP 36

why a comprehensive, multidimensional benchmarking tool?

• performance is more than – building world-class universities – enrolling students • what about equity?

• what about quality and relevance?

• what about values, attitudes and behaviors?

37

why a comprehensive, multidimensional benchmarking tool?

• institutional differentiation – different types of institutions for meeting various learning and training needs – regional engagement – technology transfer through human capital formation

• alignment of key factors

characteristics of well-performing institutions alignment of key factors Adequate Teaching and Research

Students Teaching Staff Researchers Curriculum Pedagogy Infrastructure

Graduates (skills & values) WPI

Sufficient Resources

Research

Public Budget Resources Endowment Revenues Tuition Fees Research Grants

Knowledge & Technology Transfer

Supportive Regulatory Framework Autonomy Academic Freedom Leadership Team Strategic Vision Culture of Excellence

Source: Created by Jamil Salmi

Favorable Governance

vision, leadership & reform capacity

telecommunications & digital infrastructure

resources & incentives

results attainment equity learning research technology transfer values

governance & regulatory framework

quality assurance & enhancement

diversification, articulation & information mechanisms

drivers of performance

2010

results

drivers of performance 2000

justification for conceptual framework • World Bank: Constructing Knowledge Societies (2002) • OECD Synthesis of Tertiary Education Reviews (2007) • Salmi: Challenge of Establishing World-Class Universities (2009) • Aghion et al: Governance and Performance of Research Universities (2009)

examples of indicators (results) attainment

• proportion of the workingage population (25-64) with a tertiary degree

achievement gap

• proportion from highest quintile over proportion from lowest quintile

quality

• number of ranked universities per 100,000 inhabitants

examples of indicators (results) research output

• number of citations per 100,000 inhabitants

technology transfer

• number of patents per 100,000 inhabitants

values

• proportion of voting age people who actually vote

examples of indicators (system health) regulatory framework

• legislation and QA requirements favorable to private institutions (Y/N)

institutional autonomy

• Board selects university leader (Y/N)

quality assurance

• proportion of accredited programs

examples of indicators (system health) financing

• investment in tertiary education as proportion of GDP

allocation mechanisms

• proportion of public funds allocated with performance criteria

resource utilization

• average cost of a graduate

outline of the presentation • advantages and limitations of institutional rankings • rankings of university systems: a methodological critique • benchmarking: a conceptual framework • next steps and challenges

next steps  finalizing sets of indicators  piloting in a few systems • China (national and 31 provinces) • East Asia (Vietnam, Malaysia, South Korea) • Europe and US States?

 scaling up

main challenges ahead (methodology) ďƒ˜defining and measuring multiplicity of outputs ďƒ˜finding reliable data ďƒ˜linking results and causes to be able to take action

finding the data existing databases ad-hoc surveys partnerships  countries  organizations

presenting the findings  web-based  interactive  work in progress  continuous improvement based on internal and external feedback

CHE ranking interactive website

making informed policy decisions ďƒ˜avoiding complacency

?

making informed policy decisions avoiding complacency putting things in perspective – the whole of Canada has 186 highly-cited researchers – Harvard has… 187

expansion strategy module tertiary enrollment ratei = f [SGi, PFi, (∑Eij), PSi , SAi] – high school graduation rate (SGi) – public funding for tertiary education (PFi) – distribution of enrollment among various types j of tertiary education institutions (Eij) – proportion of private sector enrollment (PSi) – student aid availability (SAi)

58

expansion strategy module tertiary enrollment ratei= αSGi + βPFi+ ∑ γjEij +δPSi + ζ SAi + εi

59

clear goals Building Minnesota’s world-leading status in the knowledge economy requires setting goals for HE and measuring results.

Governor Tim Pawlenty

benchmarking

Jamil Salmi

conclusions • different type of analysis – system lens vs. institutional – multi-dimensional – alignment

• search for actionable policy levers