Item response theory 101

ď‚Ą

How ________ is your thermometer?

ď‚Ą

Good measure is central to ability to assess well



Classical Test Theory Variance discriminate, but variance due to what (facets)?

=



Generalizability Theory Decompose variance into e.g. person, p and item, i. Identify facets causing desired, undesired variation



Item Response Theory •Probabilistic rather than deterministic •Item (question) level discrimination, not just test level •Account for person ability and item difficulty in same scale

ď‚Ą

A person should have greater probability of success on easier item than more difficult item.

ď‚Ą

A more able person should have greater probability of success on assessment items than a less able person.

ď‚Ą

Rasch analysis (in a nutshell)-how good is your test item fitting to these assertions?

(Person ability – Item Difficulty)

Personal best = 2.10m (Person ability)

Personal best = 1.72m (Person ability)

Personal best = 1.50 (Person ability)

Bar height = 1.80m (item difficulty)

What’s the probability he will clear the bar?

50:50?

Anxiety? Preparation?

Etc.

Personal best = 1.80m (Person ability)



Wright map



ICC and IIC



Polytomous IRT (e.g. partial credit model)



Fit statistics to Rasch model (chi-square)

Item 1

Item 2

Item 3

Item 1

Item 2

Item 3

Total

4.0 3.5

0.8

Information

Proportion Correct

1.0

0.6 0.4 0.2

3.0 2.5 2.0 1.5 1.0 0.5

0.0 -3

-2

-1

0 Ability

1

2

3

0.0 -3

-2

-1

0 Ability

1

2

3

Item with more than 1 mark, e.g. 4 marks Monotonic increase from 0 to 1 to 2 to 3 to 4 marks