Volume 23, No. 1, 2007 ISSN 0963-2638
Selection Selection Development Development Review Review
&
PUBLISHED BY THE BRITISH PSYCHOLOGICAL SOCIETY
SDR Competency Metrics TO MAKE THE MOST OF THE AVAILABLE selection technologies, those involved in staff assessment need to be bilingual; fluent in the languages of both competencies and psychometrics. Competency metrics provide a way of managing this interface that is reliable, predictable and accountable. The term ‘competency metrics’ refers to the alignment of personality measures with workplace competency assessments using algorithmic techniques. In this context, algorithms are mathematical operations designed to capture the relationships between various combinations and weightings of personality scales on the one hand, and behaviours associated with workplace competencies on the other. This has in the past usually been a matter of individual judgement and expertise rather than an accountable, systematic numerately literate process. In this new approach, traditional psychometrics are used to create a personality profile from an individual’s responses to personality questionnaire items. Competency metrics take you from personality profile to finely incremented competency rating; in effect, telling you to what extent an individual’s temperament facilitates or interferes with each assessed competence.
Psychometrics meet competencies The style and design of psychometric personality tests has not changed significantly over many years. Whether an instrument reflects a particular theory or model of personality, trait or type theory or, like current FFM questionnaires, is based on empirical meta-analytic studies of data generated across all these various approaches, the basic psychometric technology is pretty much the same. There is also a similarity in the nature of the output in profile form. Because the results of these questionnaires require professional interpretation, psychometric testing also supports an extensive training industry. Like pharmaceutical products, personality profiles are deemed unsafe in the hands of the un-initiated. It seems that HR professionals have been summoned to the mountain of psychometrics, rather than the mountain of psychometrics going to the Mohammed of HR. Selection & Development Review, Vol. 23, No. 1, 2007
Geoff Trickey PCL Even the seismic impact of the internet has made surprisingly little difference so far to psychometric testing. Most online assessment systems are little more than paper-and-pencil questionnaires presented on website pages. The most dramatic influence of the internet on assessment seems to have been in expanding the psychometric using population, an accelerating trend because of rapidly increasing transmission speeds and accessibility. The factor that has had a powerful influence on the nature of candidate and employee assessment is the almost universal adoption of competency frameworks for describing the behaviours necessary for effective job performance.
Addressing the variability of profile interpretation The language of competencies now provides the predominant vocabulary for describing job requirements as well as the basis for candidate assessment practices such as assessment centres and competency based interviews and appraisals. Not surprisingly, there has also been pressure for the results of psychometric assessments to be expressed in the same terms. At PCL, as at other consultancies I am sure, for many years we have been preparing individually crafted reports in which personality assessment has been shoehorned into the client’s competency framework. This labour intensive approach to the preparation of what we described as ‘Profile Match’ reports had two serious shortcomings. Firstly, they were time consuming and expensive to create. Secondly, it proved extremely difficult to control inconsistencies in personality profile interpretation and the resulting competency ratings across candidates. However, this era of hand crafting Profile Match reports gave us a proving ground within which this earlier expert judgement based process became more rigorous and systematic. The need to address these diffi3
culties provided the driving force behind the development of competency metrics and the current automated online system – PROFILE:MATCH® – which was launched last year.
Guiding principles 1. That only five factors of personality can be independently measured. Beyond this, scales are either elements within a factor (and will therefore overlap with other similar elements), or they address content from outside the personality domain (e.g. occupation interest, motivation or values items. See Hofstee, 1990). 2. That the five personality factors can account for the extensive range of workplace behaviours – in the same way that just three primary colours provide more tones and hues than can be distinguished by the human eye (Wiggins, 1979; Mount & Barrick 1995). 3. That to reflect the diversity of personality as we experience it in daily life, these five ‘primary colours of personality’ need to be combined – rather than fragmented into innumerable scales. 4. Fragments of personality scales (HICs, subthemes, etc.) are inherently less stable than complete scales and inferences cannot be drawn from them with the same degree of confidence. 5. A temperament-based competency rating may draw from any permutation of the five personality factors. Tests that pursue the fine detail of personality by proliferating scales, in our view, seem to miss this important point. 6. Overlaying or blending complete scales in various ways to achieve the fine detail of personality (including alignment of personality with competencies) builds numbers of contributing items, and achieves more robust and stable measurement.
The process We identify six steps in the process of aligning personality with competencies, although development will involve cycling back and forth through earlier steps to make refinements. Steps one and two below achieve a first approximation of personality alignment to the competency utilising the same knowledge of and experience with the dimensions of personality as inform the usual processes of profile interpretation. Clearly, our emphasis is on personality and this means 4
that the relationship with a competency defined in terms of skills, knowledge and experience must be partial. In order to address this, the competency definition has to be revised so that it is clear what is and is not being measured. The first three steps below are inferential, the final three steps cover the competency metrics. 1. The personality characteristics likely to contribute to a competency are identified (e.g. a definition of creativity is likely to include curiosity, imagination and readiness to challenge the status quo). 2. The nature of the relationship of each personality characteristic to the competency is considered (e.g. whether linear, curvilinear, and whether positive or negative). 3. The definition of each competency is revised and finalised to reflect this personality content. The remaining steps are concerned with translating these judgements into mathematical algorithms: 4. The personality/competency relationships are expressed as algorithms that will define their contribution to a competency rating. 5. The appropriate weighting of each personality scale contributing to a competency rating is estimated. 6. The composite of the contributing algorithms can then be run against a 1000+ database of personality data to evaluate distribution characteristics of the resulting competency ratings.
Summarising the benefits The development of competency metrics began as an in-house project addressing the issues we confronted in our efforts to meet client requests for assessments aligned to their competency frameworks. What began as a PCL utility has ended up as an innovative resource for other assessment practitioners. Having initially created personality assessment and competency-based report generating systems for internal PCL use, it became evident that others could operate this utility just as well as we could. The project more than met its original objectives. The intended benefits were realised and, in addition, we became aware of a number of unintended and positive consequences of the competency metric approach as we developed the online system.
Selection & Development Review, Vol. 23, No. 1, 2007
Expected benefits of competency metrics: 1. They reflect the wide adoption of competency frameworks to describe and assess workplace performance and fit well alongside assessment centres and competency-based interviews, appraisals and other organisational performance measures. 2. They allow assessment to be given a specific focus from the outset, addressing relevant and specific questions, rather than commissioning the same full personality profile for a variety of different roles. 3. They put the interpretation of personality data onto the same explicit and accountable footing as the tests that generate the personality profile. They can be researched and validated. 4. They guarantee that all candidates are treated in the same way. 5. Algorithms can be refined and developed to improve the measurement qualities, distribution characteristics and predictiveness of competency ratings, on the basis of experience and research. Unintended consequences: 6. Psychometric methods emphasise correlationbased techniques such as regression, factor analysis and principal components analysis. Non-linear relationships are invisible to correlation, yet may be crucial to performance predictions. Competency metrics readily accommodate such relationships. 7. Because competency metrics combine scales, rather than fragmenting them – inferences about performance are: (a) based on large numbers of items; and (b) are able to draw from the wide knowledge base surrounding the FFM primary scales. 8. From the user perspective, the online system operates entirely within the user friendly language of competencies. With the current system, users only need to know what they are looking for in competency terms. The psychometrics operate entirely ‘behind the scenes’. 9. While competency ratings used in assessment centres, competency based interviews or appraisals will typically be on a three-, five- or occasionally a seven-point rating scale – our competency metric technology is actually based on a scale of 45 (effectively the functioning range of the T scale), reduced – for pragmatic concerns about over-interpretation – to a 15-point scale. Selection & Development Review, Vol. 23, No. 1, 2007
Discussion Since, in effect, the original hand-crafted Profile Match reports followed the typical routine of drawing inferences from personality data on the basis of professional experience and generalisations from applied research, our group report writing experience sensitised us to the vulnerability of standard personality interpretation processes. When you have several PCL psychologists simultaneously attempting to align personality with competency frameworks on a judgement basis, the consistency problems become very apparent. How do you ensure that the emphasis and weight given to any one scale score is sufficiently similar across all report compilers – even if very tight agreements about the emphasis to be given for each scale in relation to each competency have been established? The simple fact is that you cannot – and could not – even if all the reports were to be written by the same individual. These experiences expose the weakness of the established controls in the use of personality assessments – the consistency and reliability of the questionnaires may be open to scrutiny, criticism and review, but the interpretation of the profiles being generated at the point of delivery is not. In other words, when using anything other than the most literal and simplistic personality questionnaires, one trained and experienced test user may draw different conclusions from a profile than another. While there may be valiant efforts on the part of the Society and CIPD to expose the weakness of any instrument (through reviews, training, ‘best practice’ initiatives, etc.), there is no obvious way of exposing the variability of profile interpretations based on professional judgement. Competency metrics address this issue head on. This is a methodology that puts profile interpretation onto the same accountable basis as the tests themselves. Being mathematically tied to the personality scales on which the PROFILE:MATCH® competency ratings are based, the measurement quality comes through the algorithmic process. These ratings are in fact metrics based on a T-scale and they are completely objective and completely consistent across candidates. They are also amenable to validation and research at a level of precision that individual profile interpretations are clearly not. In a recent HMRC project, we used competency metrics to align personality test scores to the seven HMRC competency definitions. 5
The overall competency ratings generated by competency metrics for project participants correlated 0.46 with ratings achieved by a selected sample considered at HMRC to exemplify the characteristics enshrined in their competency definitions.
The next step Now that PROFILE:MATCH® is up and running as an open access system, we are preparing a version for use by those professionally trained in occupational testing, PROFILE:MATCH PROTM. This will allow psychometric practitioners to use a simple user-friendly interface to develop their own competency definitions and algorithms and add their own new competencies to the competency library, as it appears within their own account area on the online system. They will, in effect, go through each of the steps originally involved in creating PROFILE:MATCH®, as described above. PROFILE:MATCH PROTM will allow practitioners to align personality assessment with the competency frameworks of their own clients with the assurance of complete consistency of competency rating across all candidates and all the other benefits of the competency metric approach.
References Hofstee, W.K.B. (1990). The use of everyday personality language for scientific purposes. European Journal of Personality, 4, 77–88. Mount, M.K. & Barrick M.R. (1995). The Big Five personality dimension: Implications for research and practice in human resource management. Research in Human Resource Management, 13, 153–200. Trickey, G. (2006). Linking personality data to HMCE’s competency framework. Strategic HR Review, 5(4). Wiggins, J.S. (1979). A psychological taxonomy of trait-descriptive terms: the interpersonal domain. Journal of Personality and Social Psychology, 37, 395–412.
The British Psychological Society and Blackwell Publishing’s unique partnership – BPS Blackwell – delivers an essential range of books for those seeking information on the latest psychology practices, research and methodology. As a BPS member, you are entitled to a 20% discount on any BPS Blackwell title. As a BPS student member, you are entitled to a 30% discount on any BPSBlackwell title.
Visit the website for further information www.bpsblackwell.co.uk
WWW.BPSBLACKWELL.CO.UK 6
Selection & Development Review, Vol. 23, No. 1, 2007