European Journal of Psychological Assessment 1/2018

Volume 34 / Number 1 / 2018

Volume 34 / Number 1 / 2018

European Journal of

Psychological Assessment

European Journal of Psychological Assessment

Editor-in-Chief Samuel Greiff Associate Editors Nicolas Becker Gary N. Burns Laurence Claes Marjolein Fokkema Penelope Hasking Dragos Iliescu Stefan Krumm Lena Lämmle Anastasiya Lipnevich René T. Proyer Ronny Scherer Eunike Wetzel Matthias Ziegler

Official Organ of the European Association of Psychological Assessment

Assessment and interventions for dementia patients and their families “This is the book on dementia that I’ve been waiting for – a concise, evidence-based volume that covers the essential aspects of diagnosis, assessment, and treatment of persons with Alzheimer’s disease and related conditions.”

Shane S. Bush, PhD, ABPP, Board Certified in Clinical Neuropsychology and Geropsychology; Coauthor of Ethical Practice in Geropsychology and coeditor of Geriatric Neuropsychology: Practice Essentials; VA New York Harbor Healthcare System, Brooklyn, NY & University of Alabama, Tuscaloosa, AL

Benjamin T. Mast / Brian P. Yochim

Alzheimer’s Disease and Dementia (Series: Advances in Psychotherapy – Evidence-Based Practice – Volume 38) 2018, viii + 76 pp.

US $29.80 / € 24.95 ISBN 978-0-88937-503-1 Also available as eBook As the number of older adults with dementia continues to skyrocket, every health care professional needs accurate, up-to-date knowledge of these conditions, their prevention, and possible treatments. This compact, evidence-based book discusses essential aspects of the diagnosis, assessment, and interventions of Alzheimer’s disease and the syndromes of dementia and mild cognitive impairment. It reviews the diagnostic criteria from the National Institute on Aging, Alzheimer’s Association, and

www.hogrefe.com

the DSM-5 and provides a broad range of treatment options, including psychosocial, educational, and lifestyle interventions.

Practitioners will especially appreciate the current overview of caregiver interventions. Practitioners and students alike will find the clear information, the tools for assessment, and other resources provided in this volume extremely useful for helping patients and their families cope with dementia.

European Journal of

Psychological Assessment Volume 34 / Number 1/ 2018 OfďŹ cial Organ of the European Association of Psychological Assessment

Editor-in-Chief

Samuel Greiff, Cognitive Science and Assessment, ECCS unit, 11, Porte des Sciences, 4366 Esch-sur-Alzette, Luxembourg (Tel. +352 46 6644-9245, E-mail samuel.greiff@uni.lu)

Editors-in-Chief (past)

Karl Schweizer, Germany (2009–2012), E-mail k.schweizer@psych.uni-frankfurt.de Matthias Ziegler, Germany (2013–2016), E-mail zieglema@hu-berlin.de

Editorial Assistant

Katharina Herborn, Cognitive Science and Assessment, ECCS unit, 11, Porte des Sciences, 4366 Esch-sur-Alzette, Luxembourg, (Tel. +352 46 6644-5578, E-mail katharina.herborn@uni.lu)

Associate Editors

Nicolas Becker, Germany; Gary N. Burns, USA/Sweden; Laurence Claes, Belgium; Marjolein Fokkema, The Netherlands; Penelope Hasking, Australia; Dragos Iliescu, Romania; Stefan Krumm, Germany; Lena Lämmle, Germany; Anastasiya Lipnevich, USA; René Proyer, Germany; Ronny Scherer, Norway; Eunike Wetzel, Germany; Matthias Ziegler, Germany

Editorial Board

Rebecca Pei-Hui Ang, Singapore Roger Azevedo, USA R. Michael Bagby, Canada Yossef S. Ben-Porath, USA Nicholas F. Benson, USA Francesca Borgonovi, France Vesna Busko, Croatia Eduardo Cascallar, Belgium Mary Louise Cashel, USA Carlo Chiorri, Italy Lee Anna Clark, USA Paul De Boeck, USA Scott L. Decker, USA Andreas Demetriou, Cyprus Annamaria Di Fabio, Italy Christine DiStefano, USA Stefan Dombrowski, USA Fritz Drasgow, USA Kadriye Ercikan, USA Rocı́o Fernández-Ballesteros, Spain Marina Fiori, France Brian F. French, USA Arthur C. Graesser, USA Patrick Griffin, Australia Jan-Eric Gustafsson, Sweden Ronald K. Hambleton, USA

Founders

Rocı́o Fernández-Ballesteros and Fernando Silva

Supporting Organizations

The journal is the official organ of the European Association of Psychological Assessment (EAPA). The EAPA was founded to promote the practice and study of psychological assessment in Europe as well as to foster the exchange of information on this discipline around the world. Members of the EAPA receive the journal in the scope of their membership fees. Further, the Division for Psychological Assessment and Evaluation, Division 2, of the International Association of Applied Psychology (IAAP) is sponsoring the journal: Members of this association receive the journal at a special rate (see below).

Publisher

Hogrefe Publishing, Merkelstr. 3, D-37085 Göttingen, Germany, Tel. +49 551 999-500, Fax +49 551 999-50111, E-mail publishing@hogrefe.com, Web http://www.hogrefe.com North America: Hogrefe Publishing, 7 Bulfinch Place, 2nd floor, Boston, MA 02114, USA, Tel. +1 866 823-4726, Fax +1 617 354-6875, E-mail customerservice@hogrefe-publishing.com, Web http://www.hogrefe.com

Production

Regina Pinks-Freybott, Hogrefe Publishing, Merkelstr. 3, D-37085 Göttingen, Germany, Tel. +49 551 999-500, Fax +49 551 999-50111, E-mail production@hogrefe.com

Subscriptions

Hogrefe Publishing, Herbert-Quandt-Strasse 4, D-37081 Göttingen, Germany, Tel. +49 551 50688-900, Fax +49 551 50688-998, E-mail zeitschriftenvertrieb@hogrefe.de

Advertising/Inserts

Melanie Beck, Hogrefe Publishing, Merkelstr. 3, D-37085 Göttingen, Germany, Tel. +49 551 999-500, Fax +49 551 999-50111, E-mail marketing@hogrefe.com

ISSN

ISSN-L 1015-5759, ISSN-Print 1015-5759, ISSN-Online 2151-2426

Copyright Information

Ó 2018 Hogrefe Publishing. This journal as well as the individual contributions and illustrations contained within it are protected under international copyright law. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without prior written permission from the publisher. All rights, including translation rights, reserved.

Publication

Published in 6 issues per annual volume (new in 2017; 4 issues from 2004 to 2016)

Subscription Prices

Annual subscription, Institutions (2018): 398.00, US $508.00; Annual subscription, Individuals (2018): 199.00, US $254.00; Postage and handling: 12.00, US $16.00; Special rates: IAAP/Colegio Oficial de Psicólogos members: 129.00, US $164.00 (+ 18.00, US $24.00 postage and handling); EAPA members: Included in membership Single issues: 66.50, US $85.00 (+ postage and handling)

Payment

Payment may be made by check, international money order, or credit card, to Hogrefe Publishing, Merkelstr. 3, D-37085 Göttingen, Germany, or, for North American customers, to Hogrefe Publishing, 7 Bulfinch Place, 2nd floor, Boston, MA 02114, USA.

Electronic Full Text

The full text of the European Journal of Psychological Assessment is available online at http://econtent.hogrefe.com and in PsycARTICLES.

Abstracting/Indexing Services

The journal is abstracted/indexed in Current Contents / Social & Behavioral Sciences (CC/S&BS), Social Sciences Citation Index (SSCI), Social SciSearch, PsycINFO, Psychological Abstracts, PSYNDEX, ERIH, and Scopus. Impact Factor (2016): 2.328.

European Journal of Psychological Assessment (2018), 34(1)

William Hanson, Canada Sven Hilbert, Germany Joeri Hofmans, Belgium Therese N. Hopfenbeck, UK Jason Immekus, USA Jan Henk Kamphuis, The Netherlands David Kaplan, USA James C. Kaufman, USA Eun Sook Kim, USA Muneo Kitajima, Japan Radhika Krishnamurthy, USA Klaus Kubinger, Austria Patrick Kyllonen, USA Kerry Lee, Singapore Chung-Ying Lin, Hong Kong Jin Liu, USA Patricia A. Lowe, USA Romain Martin, Luxembourg R. Steve McCallum, USA Helfried Moosbrugger, Germany Kevin R. Murphy, Ireland Janos Nagy, Hungary Tuulia M. Ortner, Austria Marco Perugini, Italy K. V. Petrides, UK

Aaron Pincus, USA Kenneth K. L. Poon, Singapore Ricardo Primi, Brazil Richard D. Roberts, USA Willibald Ruch, Switzerland Leslie Rutkowski, Norway Jesus F. Salgado, Spain Douglas B. Samuel, USA Manfred Schmitt, Germany Heinz Schuler, Germany Martin Sellbom, New Zealand Valerie J. Shute, USA Jonathan Templin, USA Katherine Thomas, USA Stéphane Vautier, France Fons J.R. van de Vijver, The Netherlands Michele Vecchione, Italy David Watson, USA Nathan C. Weed, USA Alina von Davier, USA Cilia Witteman, The Netherlands Moshe Zeidner, Israel Johannes Zimmermann, Germany Ada Zohar, Israel Bruno Zumbo, Canada

Ó 2018 Hogrefe Publishing

Contents Editorial

The World Beyond Rating Scales: Why We Should Think More Carefully About the Response Format in Questionnaires Eunike Wetzel and Samuel Greiff

1

Original Articles

Mindful Machine Learning: Using Machine Learning Algorithms to Predict the Practice of Mindfulness Sebastian Sauer, Ricardo Buettner, Thomas Heidenreich, Jana Lemke, Christoph Berg, and Christoph Kurz

6

Multistudy Reports

Ó 2018 Hogrefe Publishing

Construct Validity of the Eyberg Child Behavior Inventory (ECBI): An Exploratory Structural Equation Modeling Approach Silje Sommer Hukkelberg, Charlotte Reedtz, and John Kjøbli

14

You Will Attain My Goal: The Structure of Parental Goals for Children Based on an Adapted Version of the Aspirations Index Jelena Hollmann, Julia Gorges, and Elke Wild

22

Are Workaholism and Work Engagement in the Eye of the Beholder?

A Multirater Perspective on Different Forms of Working Hard Greta Mazzetti, Wilmar B. Schaufeli, and Dina Guglielmi

30

A Theoretical Approach to Resolving the Psychometric Problems Associated With the Zimbardo Time Perspective Inventory: Results From the USA, Australia, Slovenia, and the United Kingdom Frank C. Worrell, Elizabeth C. Temple, Michael T. McKay, Urška Živkovič, John L. Perry, Zena R. Mello, Bojan Musil, and Jonathan C. Cole

41

A Cross-Cultural Validation of the Implicit Positive and Negative Affect Test (IPANAT): Results From Ten Countries Across Three Continents Markus Quirin, Monika Wróbel, Andrea Norcini Pala, Stefan Stieger, Jos Brosschot, Miguel Kazén, Joshua A. Hicks, Olga Mitina, Dong Shanchuan, Ruta Lasauskaite, Nicolas Silvestrini, Patrizia Steca, Maria A. Padun, and Julius Kuhl

52

European Journal of Psychological Assessment (2018), 34(1)

Editorial The World Beyond Rating Scales Why We Should Think More Carefully About the Response Format in Questionnaires Eunike Wetzel1 and Samuel Greiff2 1

Department of Psychology, University of Mannheim, Germany

2

Cognitive Science and Assessment, University of Luxembourg, Luxembourg

Researchers constructing a new questionnaire think very carefully about a lot of things: the construct definition, the target population, the wording of the items, the item selection, and so on. These are all explicit steps in the test construction process as it is described by textbooks and research articles (Clark & Watson, 1995; Simms, 2008; Ziegler, 2014). One aspect that appears to receive less attention is the choice of response format. When the questionnaire is a self-report (or other-report) measure of one or more psychological constructs, test constructors appear to automatically implement a rating scale such as strongly disagree to strongly agree as the response format. The reason for this is that rating scales have served us well in the past and continue to do so.

However, in this editorial we will argue that choosing the response format should be an explicit step in the test construction process that deserves special attention and considerable thought. In fact, the response format should be chosen to fit the construct best. We will also argue that we need a greater diversity of response formats and more research on them.

Rating Scales: The Default Response Format in Questionnaires The most common response format in self-report or otherreport questionnaires assessing personality traits, interests, motivations, or other psychological constructs is the rating scale response format. With rating scales, each item is presented individually and respondents rate their endorsement of the item on a scale with multiple response categories (see example in Figure 1). Common rating scales include rating scales on agreement (strongly disagree to strongly agree), Ó 2018 Hogrefe Publishing

rating scales on degree or extent (not at all to very much), and frequency scales (never to always). There are a number of issues to consider when deciding which type of rating scale to use (for a comprehensive list see Saris & Gallhofer, 2014): Should it be unipolar or bipolar? How many response categories should the rating scale have? Should there be a middle (neutral) category? How should the categories be labeled (numerically, verbally, with symbols, or combinations of these)? Are the categories equidistant? Should there be a separate NA/do not want to respond category?

There is a large amount of research addressing these questions (e.g., Hernández, Drasgow, & González-Romá, 2004; Krosnick & Berent, 1993; Revilla, Saris, & Krosnick, 2014; Schwarz, Knauper, Hippler, Noelle-Neumann, & Clark, 1991). This research overwhelmingly shows that choices on these issues matter. For example, Schwarz et al. (1991) found that average ratings differed between the presentation of an 11-point rating scale with values from 0 to 10 versus values from –5 to +5 with identical verbal endpoint labels. Krosnick (1999), Clark and Watson (1995), and DeCastellarnau (2017) review some of the research and provide specific recommendations on how to deal with these matters.

Despite the consistent finding that choices on how the rating scale is set up do matter, overall rating scales work well. They allow reliable and valid assessments of a great diversity of psychological constructs and many people find them easy to use. Nevertheless, they also have a number of important drawbacks: (1) They are susceptible to response biases such as response styles and socially desirable responding.

This can have detrimental consequences such as distorting correlations between traits (Moors, 2012) or be rather inconsequential, especially when the trait and response style are unrelated or weakly correlated European Journal of Psychological Assessment (2018), 34(1), 1–5 https://doi.org/10.1027/1015-5759/a000469

2

Editorial

Figure 1. Example for the rating scale response format with items from the Big Five Triplets (Wetzel & Frick, 2017).

(Plieninger, 2017; Wetzel, Böhnke, & Rose, 2016).

Either way, from a measurement perspective, any additional influence on item responses that is not the construct of interest is problematic.1 (2) There are interindividual differences in the interpretation of response category labels. For example, going out “often” might mean once a month to one person and three times a week to another person.

(3) Different subgroups (high vs. low education level, different cultures) use rating scales differently (Johnson, Kulesa, Cho, & Shavitt, 2005; Rammstedt & Farmer, 2013).

(4) Especially with long questionnaires, rating scales may be tiresome and invite careless responding (Meade & Craig, 2012).

Variations of Rating Scales The simplest response format consists of just two response options such as true and false or yes and no. The rationale behind extending this dichotomous format to a rating scale with more than two response categories was that more information could be obtained by allowing greater differentiation between individual responses (Masters, 1988).

A number of variations or attempted advancements of rating scales have been suggested, sometimes explicitly to overcome one or more of the disadvantages of existing

1

rating scales. In some questionnaires, rather than using the same rating scale for all items, variations of the item stem that represent different degrees of the trait are used.

For example, one item in the Beck Depression Inventory-II (BDI-II; Beck, Steer, & Brown, 1996) consists of the response options 0 (= I do not feel sad), 1 (= I feel sad much of the time), 2 (= I am sad all the time), and 3 (= I am so sad or unhappy that I can’t stand it). This format might be able to reduce some problems with regular rating scales such as reducing the ambiguity of rating scale labels, but it is challenging to construct because specific behaviors need to be found that capture different trait levels while being exhaustive at the same time. In the example above, there is no response option for people who feel sad occasionally.

Another example for an attempt to improve rating scales is Visual Analogue Scales (VAS; Hayes & Patterson, 1921), which are often presented as a slider scale ranging from 0 to 100. The idea behind VAS is that, since the underlying construct is continuous, its measurement should also be continuous. However, trying to make the measurement continuous is unnecessary because methods exist for transforming a discrete (e.g., rating scale) measurement onto a continuous trait level scale (item response models, aggregating across items). In addition, VAS assume that participants can actually make such fine-grained differentiations. However, when rating scales with many (e.g., 100) numbered categories are presented to people, they tend to choose ones that are divisible by 10 (or 5), indicating that

For more information on response biases see a previous editorial by Ziegler (2015) and an overview chapter by Wetzel, Böhnke, and Brown (2016).

European Journal of Psychological Assessment (2018), 34(1), 1–5

Ó 2018 Hogrefe Publishing

Editorial

3

the potentially overtaxing differentiation is broken down into coarser segments by participants and not all available options are actually used (Henss, 1989). In line with this, research shows that adding more response categories beyond around seven does not improve measurement notably (Preston & Colman, 2000). VAS thus offer a differentiation that is more fine-grained than participants’ judgments. In consequence, these additional gradations are meaningless and should not be interpreted.

In sum, modifications of rating scales have not been able to eliminate the problems inherent to rating scales (e.g., interindividual differences in using the rating scale). Thus, replacing rating scales with alternative response formats might be a more sensible course of action than trying to improve rating scales.

Alternatives to Rating Scales

Figure 2. Example for a behaviorally anchored rating scale assessing the construct orderliness.

There appear to be few viable alternative response formats that are not just variations of the rating scale format. In the 1960s, Behaviorally Anchored Rating Scales (BARS) were proposed (Smith & Kendall, 1963). With BARS, the response options consist of behavioral anchors that represent different trait levels (see example in Figure 2). Ideally, one or two BARS would be sufficient for assessing a trait. BARS are used in particular for performance ratings in industrial and organizational psychology (Grote, 1996), but they are not widespread because their psychometric properties (e.g., reliability) were not superior to other assessment methods and their construction is complex and costly (Bernardin & Smith, 1981; Schwab, Heneman, & DeCotiis, 1975). In addition to this, finding specific behaviors that adequately represent moderate trait levels appears to be particularly challenging (Hauenstein, Brown, & Sinclair, 2010). However, with more sophisticated (e.g., item response theory) methods, it might be possible to construct BARS that provide reliable and valid assessments while simultaneously being efficient for test users. It may then be feasible to apply BARS for the assessment of other constructs such as personality traits. Attempts have been made to assess personality with concrete behavioral indicators (for an example with conscientiousness see Jackson et al., 2010) and these could be transformed into anchors on a BARS. Another alternative to rating scales is the multidimensional forced-choice (MFC) format, which has been around for a long time, but has recently gained traction with the development of item response models that allow obtaining normative trait estimates (as opposed to ipsative trait estimates) from MFC data (Brown & Maydeu-Olivares, 2011, 2013). In the MFC format, several items are presented

simultaneously to respondents in an item block and they have to rank the items according to their preference (e.g., with activities or products) or according to how well they describe the respondent (e.g., with personality items; see example in Figure 3).2 There are several variations of the MFC format that depend on the size of the item block (pairs, triplets, quads, and so forth) and the instruction to participants (full ranking vs. partial ranking). The process of responding to MFC item blocks differs from the response process to rating scale items in that the items within the block have to be weighed against each other (Sass, Frick, Reips, & Wetzel, in press). Despite the potentially higher cognitive effort involved in responding to MFC item blocks, test motivation does not appear to differ between the MFC and rating scale format (Sass et al., in press). The MFC format eliminates response styles such as extreme response style. Other response biases such as faking can still occur to some extent, though the ranking task puts a limit on the amount of faking that can occur because not all traits can be faked at the same time and equally strong (in the triplet in Figure 3, a participant would have to decide whether to focus on faking extraversion, emotional stability, or conscientiousness by placing it on rank 1). Studies have shown that the MFC format is less susceptible to faking than the rating scale format (Christiansen, Burns, & Montgomery, 2005). Constructing an MFC questionnaire, however, is more complex than constructing a rating scale questionnaire because a lot of additional considerations play a role such as which items are presented in a block, and more research is needed on these basic test construction issues.

2

The MFC format therefore is both an item format and a response format at the same time.

Ă“ 2018 Hogrefe Publishing

European Journal of Psychological Assessment (2018), 34(1), 1–5

4

Figure 3. Example for the multidimensional forced-choice format from the Big Five Triplets (Wetzel & Frick, 2017).

Concluding Remarks When writing this editorial, we realized that there are surprisingly few real alternatives to the rating scale format.

In contrast, in achievement testing, there are numerous creative assessment methods such as computer-simulated microworlds (Wüstenberg, Greiff, & Funke, 2012) or innovatively constructed response formats (Thissen, Koch, Becker, & Spinath, 2016) that complement classical openended questions and multiple-choice formats.

In a way, one could ask, why is assessing noncognitive traits so boring and why do we always use rating scales?

One reason appears to be that rating scales overall do the job they are supposed to do, despite their disadvantages, especially when we are only looking at homogenous student samples from one country. With more heterogeneous samples and in particular in cross-cultural research, this might not be the case. Another reason for the popularity of rating scales might be that they are very convenient and easy to construct. Thus, we as test constructors have not found it necessary to look beyond them. In addition, some of the attempts to introduce alternatives have run into severe problems (see BARS or formerly the ipsativity of MFC trait scores). However, some of these problems can be solved with more sophisticated methods such as item response theory as in the case of MFC data.

This editorial is not a call to stop using rating scales, but we believe that the choice of response format should be an explicit step in the process of constructing a questionnaire.

Therefore, we would like to encourage researchers and test constructors to explicitly consider alternatives to rating scales instead of automatically using a rating scale and to make a well-thought-out decision that is based on weighing the pros and cons. Some feasible alternatives such as the MFC format already exist, but more are needed. We encourage research on response formats, both research on existing response formats and especially research

European Journal of Psychological Assessment (2018), 34(1), 1–5

Editorial

exploring alternative response formats. Right now, many researchers including the authors of this editorial assume that the default of rating scales is appropriate for virtually any construct we assess with self-report questionnaires and the only aspect that needs to be considered is the specifics of the rating scale (e.g., number of response categories). However, this may not be the case. Different response formats may be appropriate for different constructs, as in achievement testing. For example, there are a number of constructs that often show low response variability when assessed with rating scales (test motivation, self-esteem). In these cases, a different response format (perhaps with specific behavioral anchors) might be more informative. In psychological assessment, it is our goal to measure the constructs we are interested in as good as we can and to draw valid inferences regarding people’s trait levels from our questionnaire results. Thus, we need to make sure we use the response format that will allow achieving these goals. Test constructors think so carefully about many details of the questionnaire they are constructing (e.g., the wording of individual items or which items to select for the final version). They should think equally carefully about which response format to use.

Acknowledgment The authors thank Susanne Frick for her comments on a draft of this editorial.

References Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Beck Depression Inventory-II. San Antonio, TX: The Psychological Corporation.

Bernardin, H. J., & Smith, P. C. (1981). A clarification of some issues regarding the development and use of behaviorally anchored rating-scales (BARS). Journal of Applied Psychology, 66, 458–463. https://doi.org/10.1037/0021-9010.

66.4.458 Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460–502. https://doi.org/ 10.1177/0013164410375112 Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires.

Psychological Methods, 18, 36–52. https://doi.org/10.1037/ a0030641 Christiansen, N. D., Burns, G. N., & Montgomery, G. E. (2005).

Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18, 267–307. https://doi.org/10.1207/s15327043hup1803_4 Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309–319. https://doi.org/10.1037/1040-3590.7.3.309 DeCastellarnau, A. (2017). A classification of response scale characteristics that affect data quality: A literature review.

Quality & Quantity. Advance online publication. https://doi.org/ 10.1007/s11135–017-0533–4

Ó 2018 Hogrefe Publishing

Editorial

Grote, D. (1996). The complete guide to performance appraisal.

New York, NY: American Management Association.

Hauenstein, N. M. A., Brown, R. D., & Sinclair, A. L. (2010). BARS and those mysterious, missing middle anchors. Journal of Business and Psychology, 25, 663–672. https://doi.org/ 10.1007/s10869-010-9180-7 Hayes, M. H. S., & Patterson, D. G. (1921). Experimental development of the graphic rating method. Psychological Bulletin, 18, 98–99.

Henss, R. (1989). Zur Vergleichbarkeit von Ratingskalen unterschiedlicher Kategorienzahl [On the comparability of rating scales with different numbers of categories]. Psychologische Beiträge, 31, 264–284.

Hernández, A., Drasgow, F., & González-Romá, V. (2004). Investigating the functioning of a middle category by means of a mixed-measurement model. Journal of Applied Psychology, 89, 687–699. https://doi.org/10.1037/0021-9010.89.4.687 Jackson, J. J., Wood, D., Bogg, T., Walton, K. E., Harms, P. D., & Roberts, B. W. (2010). What do conscientious people do?

Development and validation of the Behavioral Indicators of Conscientiousness (BIC). Journal of Research in Personality, 44, 501–511. https://doi.org/10.1016/j.jrp.2010.06.005 Johnson, T., Kulesa, P., Cho, Y. I., & Shavitt, S. (2005). The relation between culture and response styles – Evidence from 19 countries. Journal of Cross-Cultural Psychology, 36, 264–277. https://doi.org/10.1177/0022022104272905 Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://doi.org/10.1146/annurev. psych.50.1.537 Krosnick, J. A., & Berent, M. K. (1993). Comparisons of party identification and policy preferences – The impact of survey question format. American Journal of Political Science, 37, 941–964. https://doi.org/10.2307/2111580 Masters, G. N. (1988). Measurement models for ordered response categories. In R. Langeheine & J. Rost (Eds.), Latent trait and latent class models (pp. 11–29). New York, NY: Plenum Press.

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455. https://doi.org/10.1037/A0028085 Moors, G. (2012). The effect of response style bias on the measurement of transformational, transactional, and laissezfaire leadership. European Journal of Work and Organizational Psychology, 21, 271–298. https://doi.org/10.1080/1359432x.

2010.550680 Plieninger, H. (2017). Mountain or molehill? A simulation study on the impact of response styles. Educational and Psychological Measurement, 77, 32–53. https://doi.org/10.1177/ 0013164416636655 Preston, C. C., & Colman, A. M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1–15. https://doi.org/10.1016/S0001-6918 (99)00050-5 Rammstedt, B., & Farmer, R. F. (2013). The impact of acquiescence on the evaluation of personality structure. Psychological Assessment, 25, 1137–1145. https://doi.org/10.1037/ a0033323 Revilla, M. A., Saris, W. E., & Krosnick, J. A. (2014). Choosing the number of categories in agree-disagree scales. Sociological Methods & Research, 43, 73–97. https://doi.org/10.1177/ 0049124113509605 Saris, W. E., & Gallhofer, I. N. (2014). Design, evaluation, and analysis of questionnaires for survey research. Hoboken, NJ: Wiley.

Sass, R., Frick, S., Reips, U.-D., & Wetzel, E. (in press). Taking the test-taker’s perspective: Response process and test motivation

Ó 2018 Hogrefe Publishing

5

in multidimensional forced-choice vs. rating scale instruments.

Assessment.

Schwab, D. P., Heneman, H. G., & DeCotiis, T. A. (1975). Behaviorally anchored rating scales: A review of the literature.

Personnel Psychology, 28, 549–562. https://doi.org/10.1111/ J.1744-6570.1975.Tb01392.X Schwarz, N., Knauper, B., Hippler, H. J., Noelle-Neumann, E., & Clark, L. (1991). Rating scales – Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570–582. https://doi.org/10.1086/269282 Simms, L. J. (2008). Classical and modern methods of psychological scale construction. Social and Personality Psychology Compass, 2, 414–433. https://doi.org/10.1111/j.1751-9004.

2007.00044.x Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149–155. https://doi.org/10.1037/H0047060 Thissen, A., Koch, M., Becker, N., & Spinath, F. M. (2016).

Construct your own response: The cube construction task as a novel format for the assessment of spatial ability. European Journal of Psychological Assessment. Advance online publication. https://doi.org/10.1027/1015–5759/a000342 Wetzel, E., Böhnke, J. R., & Brown, A. (2016). Response biases. In F. R. L. Leong, B. Bartram, F. Cheung, K. F. Geisinger, & D. Iliescu (Eds.), The ITC international handbook of testing and assessment (pp. 349–363). New York, NY: Oxford University Press.

Wetzel, E., Böhnke, J. R., & Rose, N. (2016). A simulation study on methods of correcting for the effects of extreme response style.

Educational and Psychological Measurement, 76, 304–324. https://doi.org/10.1177/0013164415591848 Wetzel, E., & Frick, S. (2017). The Big Five Triplets – Development of a multidimensional forced-choice questionnaire. Manuscript in preparation.

Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving – More than reasoning? Intelligence, 40, 1–14. https:// doi.org/10.1016/j.intell.2011.11.003 Ziegler, M. (2014). Stop and state your intentions! Let’s not forget the ABC of test construction. European Journal of Psychological Assessment, 30, 239–242. https://doi.org/10.1027/1015-5759/ a000228 Ziegler, M. (2015). “F*** you, I won’t do what you told me!” – Response biases as threats to psychological assessment.

European Journal of Psychological Assessment, 31, 153–158. https://doi.org/10.1027/1015-5759/a000292

Eunike Wetzel Department of Psychology University of Mannheim L13, 15 68161 Mannheim Germany eunike.wetzel@uni-mannheim.de

Samuel Greiff Cognitive Science and Assessment University of Luxembourg 6, rue Richard Coudenhove-Kalergi 4366 Esch-sur-Alzette Luxembourg samuel.greiff@uni.lu

European Journal of Psychological Assessment (2018), 34(1), 1–5

Original Article

Mindful Machine Learning Using Machine Learning Algorithms to Predict the Practice of Mindfulness Sebastian Sauer,1,2 Ricardo Buettner,1 Thomas Heidenreich,3 Jana Lemke,4 Christoph Berg,1 and Christoph Kurz5 1

Institute of Business Psychology/ Institute of Management & Information Systems, FOM University of Applied Sciences,

Munich, Germany 2

Brain, Mind, and Healing Program, Samueli Institute, Alexandria, VA, USA

3

Faculty of Social Work, Health, and Nursing, Esslingen University of Applied Sciences, Germany

4

Institute of Transcultural Health Sciences, Viadrina University, Frankfurt/Oder, Germany

5

Institute of Health Economics and Health Care Managemant, Helmholtz Zentrum München, Germany

Abstract: Mindfulness refers to a stance of nonjudgmental awareness of present-moment experiences. A growing body of research suggests that mindfulness may increase cognitive resources, thereby buffering stress. However, existing models have not achieved a consensus on how mindfulness should be operationalized. As the sound measurement of mindfulness is the foundation needed before substantial hypotheses can be supported, we propose a novel way of gauging the psychometric quality of a mindfulness measurement instrument (the Freiburg Mindfulness Inventory; FMI). Specifically, we employed 10 predictive algorithms to scrutinize the measurement quality of the FMI. Our criterion of measurement quality was the degree to which an algorithm separated mindfulness practitioner from nonpractitioners in a sample of N = 276. A high predictive accuracy of class membership can be taken as an indicator of the psychometric quality of the instrument. In sum, two findings are of interest. First, over and above some items of the FMI were able to reliably predict class membership. However, some items appeared to be uninformative. Second, from an applied methodological point of view, it appears that machine learning algorithms can outperform traditional predictive methods such as logistic regression. This finding may generalize to other branches of research.

Keywords: machine learning, predictive modeling, mindfulness, measurement, questionnaire

The idea that mindfulness may be able to alleviate a variety of psychosomatic symptoms has gained widespread currency in recent years (Chiesa, Calati, & Serretti, 2011; Hofmann, Sawyer, Witt, & Oh, 2010). Mindfulness as a state of consciousness may be characterized by an attentional stance that focuses on the present moment (“presence”) with a nonjudgmental attitude toward all perceptions and sensations (“acceptance”) that may be part of the actual experience (Sauer, Walach, et al., 2013). It implies a sense of being in the present, underlining the vivid awareness of sensory and mental experience, and calls for a voluntarily disruption of automatic mental processes of cognitive evaluation such as “good,” “bad,” “wanted,” or “unwanted,” as well as the suspension of labeling experience with emotional tones.

Several meta-analytical reviews have investigated the relations between mindfulness, health, and well-being and have corroborated the health-relevant effects of mindfulness (Eberth & Sedlmeier, 2012; Sedlmeier et al., 2012; Zainal, Booth, & Huppert, 2013). Symptoms for which health-relevant effects of mindfulness have been found European Journal of Psychological Assessment (2018), 34(1), 6–13 DOI: 10.1027/1015-5759/a000312

beneficial include depression, anxiety, stress, pain, substance abuse, personality disorders, and elevated blood pressure, among others. Results from neurological research have begun to reveal the biological basis of mindfulness and have linked mindfulness back to several brain regions known to be involved in attention and emotion-regulation processes. These brain areas include the cingulate cortex or the insula (Farb, Anderson, & Segal, 2012; Tang & Posner, 2013; Teper & Inzlicht, 2013). A recent review suggests that mindfulness may buffer age-related neural decay, thus preventing the brain from aging prematurely (Gard, Hölzel, & Lazar, 2014).

In the light of mindfulness being a promising area of psychological research, rigorous examination is in place for scrutinizing the claims that mindfulness-based health interventions have the potential to improve health. One condition to test the health-related effects of mindfulness is that it can be validly measured. To date, more than a dozen instruments have been published, apparently confirming the notion that mindfulness can be measured (Baer, 2011; Sauer, Walach, et al., 2013). Ó 2016 Hogrefe Publishing

S. Sauer et al., Mindful Machine Learning

Yet, no consensus on how mindfulness should be operationalized has emerged out of this variety of instruments.

On the contrary, severe criticism of the common approaches has been put forth (Grossman, 2008, 2011).

One criticism hinges on the fact that the instruments build on diverging, even contradicting ideas of mindfulness.

In the light of this, it is the aim of the current research to further the measurement of mindfulness. To this end, we suggest that the methodology that is currently employed, as sophisticated as it may be, is not sufficient and should be complemented by novel methods. Machine learning may provide such a method that is novel to psychometrics.

Machine learning is an umbrella term for a dynamic array of algorithms for predictive modeling that can supplement more “traditional” statistical approaches. Albeit not recognized much in psychology to date, in other fields such as gene expression, machine learning algorithms are now being considered a standard instrument (Dudoit, Fridlyand, & Speed, 2002; Shipp et al., 2002; Ye et al., 2003).

It is still a subject of debate as to which of the many algorithms can be considered superior to others, or, strictly speaking, which algorithms fit with which problems and data. A practical recommendation (Hastie, Tibshirani, & Friedman, 2009) is, when in doubt, to employ several algorithms and to aggregate the results. This approach was taken in the present paper.

More specifically, we used 10 different predictive algorithms to test whether the items from a well-known mindfulness inventory would be able to predict a behavioral criterion, that is, whether the participants were practicing mindfulness on a regular basis or not. This correspondence hinges on the rationale that individuals undergoing regular mindfulness training should show a higher mindfulness trait level compared to individuals who do not. Thus, we strive to improve the measurement of mindfulness by testing the predictive validity of mindfulness items. Testing the psychometric validity using external behavioral criteria is by no means new. However, we used a variety of machine learning algorithms, an approach that is novel to the area of mindfulness and to many other areas of psychology, as well.

In sum, our approach consists of testing the psychometric quality of one mindfulness instrument using a method novel to psychometrics, that is, machine learning techniques. Our first research question is whether the items of the instruments predict a behavioral criterion (i.e., regular mindfulness practice). The degree to which the items correctly identify class membership (practice yes/no) is taken as an indicator of the psychometric quality of the item. Secondly, we were interested in whether the machine learning algorithms would outperform more “classical” methods such as logistic regression. Thus, we contrasted the machine learning algorithms with regression as a classical Ó 2016 Hogrefe Publishing

7

method in psychometrics in order to evaluate the purported predictive advantage of machine learning.

Current Mindfulness Measurement Practices Utilizing psychometric scales is the standard approach for measuring mindfulness. Consequently, a substantial and growing number of scales have been published (Baer, 2011; Sauer, Walach, et al., 2013). The advantages of the psychometric approach are straightforward: convenient and quick application, well-known methodology, and empirical support (Mars & Abbey, 2010).

At the same time, there are shortcomings that may place considerable limits on the validity of the psychometric method (Grossman, 2008). One argument is that a wide array of conflicting definitions exist. For example, the concept of mindfulness put forward by Langer and Moldoveanu (2000) differs substantially from Brown and Ryan (2003) or from Kabat-Zinn (Bishop et al., 2004), to name some of the most influential researchers. As a result of different conceptions and theoretical backgrounds, the questionnaires differ greatly. The same is true for the number of supposed factors and the aspects deemed central to the construct (Grossman, 2011). The most widely used psychometric scales include the Five Factor Mindfulness Inventory (FFMQ; Bohlmeijer, Ten Klooster, Fledderus, Veehof, & Baer, 2011), the Mindfulness Attention and Awareness Scale (MAAS; Brown & Ryan, 2003), and the Freiburg Mindfulness Inventory (FMI; Kohls, Sauer, & Walach, 2009). We chose the FMI because it is a well-known instrument that combines two features. First, it incorporates both attentional and emotional components in the construct of mindfulness as opposed to the MAAS, for example. Second, it is shorter than some other instruments; for example, the FFMQ consists of 39 items.

Method Sample For increased sample size, we included two samples in this study; total sample size was N = 276. As Sample 2 was comparatively small, we refrained from analyzing each sample individually and report the combined results only. Instead, we used a statistical procedure (see below) as replication procedure rather than replicating the analysis on the second sample. Data can be accessed in Electronic Supplementary Material (ESM) 5.

Sample 1 Sample 1 (N = 201) was collected as part of an unpublished online study investigating the relations between European Journal of Psychological Assessment (2018), 34(1), 6–13

8

mindfulness, health, and emotion. About two thirds of the sample (n = 129) did not indicate that they engaged in any mindfulness training, whereas n = 72 individuals (36%) reported practicing mindfulness on a regular basis.

Examples of types of mindfulness training included Buddhist meditation or Yoga. Their mean ages were 35 years for nonpractitioners (SD = 13) and 39 years for practitioners (SD = 11); n = 141 (70%) persons were female and n = 59 (29%) persons were male (one missing value). The inclusion criterion for the mindfulness group was “regular mindfulness meditation practice.”

Sample 2 Sample 2 (N = 75) was comprised of one half (n = 38) expert mindfulness practitioners (21 female, 17 male) and an ageand sex-matched half of (n = 37) nonmindfulness practitioners (28 female, 10 male). Practitioners were trained in different Buddhist mindfulness traditions. The mean age was 51 years in both groups (SD = 10 years). Care was taken to obtain distinct groups (i.e., mindfulness practitioners vs. nonpractitioners). The inclusion criteria were at least 5 years of ongoing mindfulness practice. See Sauer et al.

(2012) for details.

Analytical Procedure The analytical procedure consisted of predicting class membership (mindfulness practitioners vs. nonpractitioners) using 10 different algorithms. The predictive accuracy (see below) was taken as an indicator of the psychometric quality of the mindfulness measurement instrument.

One of the dangers of data modeling is overfitting. Overfitting refers to a situation in which a statistical model is excessively complex and thereby describes random noise as a systematic pattern. A possible solution to this problem is to split the data into a training set and a test set (Hastie et al., 2009). The model is then built on the training set only; for model results, the test set is used. We employed a more stringent variant known as the repeated k-fold cross-validation procedure (Kuhn, 2008). Here, the data are split into k distinct blocks of similar size. In our case, we split the data into k = 10 blocks and repeated this procedure for n = 10 repetitions. Then, in n runs, each of the k blocks was left out, and the model was fit to the remaining 9/10 of the sample. The model results, built on the “left out” 1/10, were then averaged.

All algorithms were taken from the R package caret (Kuhn, 2008). The statistical software R 3.0.1 on a Windows 7 computer was used as the statistical environment. Item scores were standardized to fall within the interval of 0 to 1. We used the default values of the statistical procedures (see R syntax for details, ESM 4).

To gauge the predictive quality of the mindfulness instrument, we computed the accuracy of predicting the class European Journal of Psychological Assessment (2018), 34(1), 6–13

S. Sauer et al., Mindful Machine Learning

membership. More specifically, we split the predictive accuracy into the two aspects of sensitivity and specificity (Altman & Bland, 1994). Specificity is defined as the rate of mindfulness practitioners correctly identified as such (“true positives”). On the other hand, specificity refers to the rate of correctly identified nonpractitioners (“true negatives”). In addition, we computed Cohen’s Kappa (Fleiss & Cohen, 1973) as this statistic corrects for chance concordance. For this statistic, 0 indicates no concordance between predicted and actual values. According to Landis and Koch (1977), .41 < κ < .60 can be seen as moderate concordance, with smaller values indicating only slight or poor concordance and higher values indicating substantial concordance.

Instruments The psychometric instrument under investigation was the Freiburg Mindfulness Inventory short version (FMI; Kohls et al., 2009). The instrument was designed to measure mindfulness as a stable trait. A trait refers to a personality aspect consistent in various situations and stable for at least some weeks (Larsen & Buss, 2008). Whereas the scale was first developed to be a unidimensional scale, a recent study indicated a two-factor solution (Kohls et al., 2009). It has been validated not only by classical psychometric methods such as exploratory and confirmatory factor analyses but also using item response theory (Sauer, Walach, Offenbächer, Lynch, & Kohls, 2011). The instrument in its current form consists of 14 items with four answer options.

The instrument has been employed in a number of studies (Eisendrath et al., 2008; Leigh, Bowen, & Marlatt, 2005; Sauer, Walach, & Kohls, 2010).

Predictive Models The following 10 models were employed: Generalized linear model (glm; i.e., logistic regression), Neural networks (nn), Boosted classification trees (ada), Extreme learning machines (elm), Stochastic Gradient Boosting machine (gbm), k nearest neighbors (knn), Quadratic discriminant analysis (qda), Random Forests (rf ), Support vector machine with linear kernel (SvmLinear), and Support vector machine with polynomial kernel function (svmPoly). We chose models based on their predictive track record (see ESM 1 for details).

Results As expected, the overall mindfulness level of the mindfulness practitioner group was substantially higher than that Ó 2016 Hogrefe Publishing

S. Sauer et al., Mindful Machine Learning

9

14 13 12 11 10

Item

9 8 7 6 5 4 3 2 1 0.00

0.25

0.50

0.75

Cohen d Figure 1. Cohen’s d (mindfulness trait level differences) for each item. This figure shows the effect size difference (Cohen’s d) in individual mindfulness levels between mindfulness practitioners and nonpractitioners.

of the nonpractitioners (the average across all items; see Figure C1 in ESM 3). Mindfulness trait levels differed statistically significantly between the practitioners and nonpractitioners, respectively, t(241.61) = 7.37, p < .001, d = .85 (large effect; 1.13 < d < .56). The picture was similar at the item level (cf. Figure C2). Figure 1 shows the effect size of the differences in mindfulness levels between the two groups for each item. The items that separated the mindfulness practitioners most strongly from the nonpractitioners in terms of Cohen’s d were items 1, 5, and 14 (medium to strong effect sizes). Items 2, 10, and 12 showed a small effect size. Descriptive statistics for all items (Table B1), as well as a visualization of the correlation matrix (Figure C3), can be found in the ESM 2 and 3, respectively.

For the predictive models, the mean sensitivity across all 10 models was good (M = .76; Mdn = .81; SD = 0.24). The mean specificity was acceptable (M = .67; Mdn = .70; SD = 0.23. These values must be judged against the baseline rate of nonpractitioners of .60. Thus, the ratios were 1.27 (average sensitivity to baseline) and 1.12 (average specificity to baseline). All models except ada had sensitivity values above the baseline. All models except ada and elm had specificity values above the baseline (see Figure 2).

The best algorithm in terms of accuracy was, by far, the rf algorithm (sensitivity = .99; specificity = 1). For Cohen’s kappa, the picture was similar. The mean kappa was moderate in size (M = .44; Mdn = .50). Interestingly, the range of predictive accuracy [max – min] of the 10 algorithms differed substantially (range for sensitivity: .88; range for Ó 2016 Hogrefe Publishing

specificity: .76). It seems clear that relying on a single algorithm could lead to poor predictive performance. However, the variability (range) for accuracy was primarily due to one algorithm, which showed poor performance (ada). In sum, the data provided support for our hypothesis that using an aggregated accuracy statistic would be more adequate than relying on one algorithm alone. The machine learning algorithms outperformed the “classical” models (i.e., logistic regression, Figure 3), but only if ada was excluded.

As the rf algorithm allows to gauge the relative predictive importance of the items and because this model was the most accurate, it is interesting to know the importance of the items as computed by this algorithm (see details in ESM 1). The five items with the highest importance were 3, 2, 10, 12, and 8. Interestingly, the items with the lowest values (1, 6, 7, 9, and 11) did not include Item 13, which had been flagged as problematic in previous research (Sauer, Ziegler, Danay, Ives, & Kohls, 2013).

Discussion In this paper we tested the predictive quality of the FMI in an attempt to gauge the validity of the scale using machine learning algorithms. This novel approach was applied to complement existing approaches of the measurement of mindfulness. In sum, our results indicate that the FMI is able to distinguish mindfulness practitioners from European Journal of Psychological Assessment (2018), 34(1), 6–13

10

S. Sauer et al., Mindful Machine Learning

svmPoly svmLinear

rf

Model

qda

Coefficient

nn

sensitivity specificity kappa

knn glm gbm elm ada −0.5

0.5

0.0

1.0

Value Figure 2. Results of predictive models. This figure shows sensitivity, specificity, and the kappa coefficient for each of the 10 predictive models.

classical

ML

ML−wo.ada

0.8

value

0.6

0.4

0.2

Figure 3. Comparison of predictive strengths of different types of models.

The figure shows the predictive strength of (logistic) regression (glm) as a “classical” model (a), machine learning (ML) models (b), and ML models without ada (ML-wo.ada), a model which performed exceptionally weak (c). The values (i.e., sensitivity, specificity, kappa) of the ML models are based on the average performance of the rest of the models presented in this research.

0.0 sensitivity specificity kappa mean.all sensitivity specificity kappa mean.all sensitivity specificity kappa mean.all

variable

nonpractitioners. This result can be taken as support for the FMI’s criterion validity. However, substantial room for improvement exists. The contribution of the items was not homogeneous, and the importance of many items was weak.

The analytical method employed in this research relied on three aspects. First, we used well-acknowledged predictive algorithms novel to psychometrics. Second, we used a cross-validation approach (repeated k-fold cross-validation).

Third, our results did not rely on only one algorithm but rather on the average of several (i.e., 10) algorithms, and thus can probably be regarded as stable.

With regard to our first research question, one can infer that the FMI items succeeded in predicting class membership. Our results support the predictive quality of the items; however, substantial room for improvement exists. Of

European Journal of Psychological Assessment (2018), 34(1), 6–13

particular interest is that only a few items substantially contributed toward separating the two groups (mindfulness practitioners vs. nonpractitioners). This result substantiates a similar notion regarding items from a different mindfulness questionnaire (Hoefling, Moosbrugger, SchermellehEngel, Heidenreich, & Höfling, 2011; Van Dam, Earleywine, & Borders, 2010). Item 13, which was flagged as problematic in previous research (Sauer, Ziegler, et al., 2013), was not among the most problematic items in the current study.

However, given the argument that negatively coded items do not appear to simply convey the “same but reversed” effect, we propose that Item 13 still be treated as a revision candidate (Hoefling et al., 2011).

With regard to our second research question, we conclude that machine learning methods can outperform more classical approaches. In more detail, predictive algorithms

Ó 2016 Hogrefe Publishing

S. Sauer et al., Mindful Machine Learning

differ in the accuracy with which they are able to separate mindfulness practitioners from nonpractitioners. Due to the particularities of the analytical problem and its data, some algorithms may obviously be severely misguided. In our case, ada, generally known as a high-performing algorithm, performed poorly. Whereas we can only speculate about the statistical reasons behind the performance problems, this result supports the hypothesis that aggregating algorithms is generally superior to relying on one standard algorithm alone, at least if the algorithm has no prior experience with the specific situation. On the other hand, rf performed extremely well, and the “classic psychological” algorithm, logistic regression (glm), yielded a somewhat intermediate performance. This is particularly interesting as psychological research makes wide use of regression analysis (Cohen, Cohen, West, & Aiken, 2013). Contrary to the customary practice, our results suggest that algorithmic pluralism is superior to algorithmic primacy. Note that the presented results rely on the replication results of the test sample.

To date, we can only speculate as to why rf performed so well (nearly all persons were correctly classified), whereas ada performed poorly (worse classification as expected by chance). A reason for this finding may be that ada penalizes wrong classification and puts emphasis on wrong classified cases in following rounds. So one may speculate that some participants were “odd” in the sense that they were not classifiable. Those persons have may scored highly on relevant items without pursuing mindfulness training. A reason for the success of rf may be that rf is robust as it is built on a repeated sampling rationale (both for variables and cases). As a baseline, this finding demonstrates that several models should be tested, and, if in doubt, averaged.

More research is clearly needed. We recommend that future research similarly compute several models and compare the results.

Some words of caution need to be presented. First, one limitation of this study is that the quality of the online sample was limited. Limited sample quality may be likely to attenuate existing associations and the power of statistical tests. In addition, the allocation to groups in the online sample relied on self-report and was not controlled. Third, the predictive criterion (mindfulness practice or not) is only one of many plausible criteria. We would like to emphasize that this criterion is probably unable to produce a pure splitting of the groups by their mindfulness levels. The reason is that mindfulness level is less than perfectly correlated with (amount of mindfulness) practice. More specifically, we have not distinguished quantity (how much training) nor quality (what type of training) of mindfulness training. We would expect that differentiating quantity and quality of training will make the connection to mindfulness trait level even clearer. Furthermore, our operationalization of mindfulness was narrow (FMI only). Different instruments Ó 2016 Hogrefe Publishing

11

should be scrutinized to gauge the value of machine learning for mindfulness measurement.

The results and the limitations of the present research provide some ideas for new avenues of research, four of which we would like to spell out. (i) We urge researchers to make use of the predictive models that are widely used in other high-quality research branches such as gene expression or molecular biology in general. These results are not only of importance for the research on mindfulness but may help researchers in different psychological areas as well. It should be mentioned that the R package caret provides a unified (and simple) syntax for all of its approximately 180 models, thus providing comfortable and lowbarrier access to researchers who do not have a background in machine learning (cf. Kuhn, 2008). We recommend selecting a bunch of methods as we did and to average the result. Random forests should definitely be among the selected algorithms. (ii) In a similar vein, we propose that different criteria be used as the target to be predicted by actual machine learning algorithms. Although mindfulness practice is perhaps one of the most straightforward criteria to choose from, other criteria such as drug intake or health incidents may provide empirically founded criteria (Leigh et al., 2005). (iii) In addition, it seems strongly warranted that future research strives to improve the instrument’s items. Only some items contributed to group differentiation. The rest of the items should be considered as revision candidates. (iv) Many instruments for measuring mindfulness exist; their psychometric property should also be investigated using machine learning procedures given the successful application of the method in this study.

For the time being, we conclude that a multimodel predictive approach is advisable. Machine learning methods are useful for testing certain aspects of the validity of measurement instruments in psychology and in mindfulness research in particular. Given the widespread interest in and the promising preventative and curative health benefits of mindfulness, the measurement of mindfulness should be further investigated by means of machine learning techniques.

Electronic Supplementary Material The electronic supplementary material is available with the online version of the article at http://dx.doi.org/10.1027/ 1015-5759/a000312 ESM 1. List of predictive models (text).

Description of the statistical rationale of the predictive models employed in the present study.

ESM 2. Table (text).

Table reporting detailed descriptive statistics.

ESM 3. 3 Figures (PDF).

Additional figures.

European Journal of Psychological Assessment (2018), 34(1), 6–13

12

ESM 4. R file.

R Syntax ESM 5. Data (csv format).

Data of sample and replication.

References Altman, D. G., & Bland, J. M. (1994). Diagnostic tests. 1: Sensitivity and specificity. BMJ: British Medical Journal, 308, 1552.

Baer, R. A. (2011). Measuring mindfulness. Contemporary Buddhism, 12, 241–261.

Bishop, S. R., Lau, M., Shapiro, S., Carlson, L., Anderson, N. D., Carmody, J., . . . Speca, M. (2004). Mindfulness: A proposed operational definition. Clinical Psychology Science and Practice, 11, 230–241.

Bohlmeijer, E., Ten Klooster, P. M., Fledderus, M., Veehof, M., & Baer, R. (2011). Psychometric properties of the five facet mindfulness questionnaire in depressed adults and development of a short form. Assessment, 18, 308–320. doi: 10.1177/ 1073191111408231 Brown, K. W., & Ryan, R. M. (2003). The benefits of being present: Mindfulness and its role in psychological well-being. Journal of Personality and Social Psychology, 84, 822–848. doi: 10.1037/ 0022-3514.84.4.822 Chiesa, A., Calati, R., & Serretti, A. (2011). Does mindfulness training improve cognitive abilities? A systematic review of neuropsychological findings. Clinical Psychology Review, 31, 449–464. doi: 10.1016/j.cpr.2010.11.003 Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2013). Applied multiple regression/correlation analysis for the behavioral sciences. London, UK: Routledge.

Dudoit, S., Fridlyand, J., & Speed, T. P. (2002). Comparison of discrimination methods for the classification of tumors using gene expression data. Journal of the American Statistical Association, 97, 77–87. doi: 10.1198/ 016214502753479248 Eberth, J., & Sedlmeier, P. (2012). The effects of mindfulness meditation: A meta-analysis. Mindfulness, 3, 174–189. doi: 10.1007/s12671-012-0101-x Eisendrath, S. J., Delucchi, K., Bitner, R., Fenimore, P., Smit, M., & McLane, M. (2008). Mindfulness-based cognitive therapy for treatment-resistant depression: A pilot study. Psychotherapy and Psychosomatics, 77, 319–320.

Farb, N. A. S., Anderson, A. K., & Segal, Z. V. (2012). The mindful brain and emotion regulation in mood disorders. Canadian Journal of Psychiatry. Revue Canadienne de Psychiatrie, 57, 70–77.

Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619.

Gard, T., Hölzel, B. K., & Lazar, S. W. (2014). The potential effects of meditation on age-related cognitive decline: A systematic review. Annals of the New York Academy of Sciences, 1307, 89–103. doi: 10.1111/nyas.12348 Grossman, P. (2008). On measuring mindfulness in psychosomatic and psychological research. Journal of Psychosomatic Research, 64, 405–408.

Grossman, P. (2011). Defining mindfulness by How poorly I think I pay attention during everyday awareness and other intractable problems for psychology’s (re)invention of mindfulness: Comment on Brown et al. (2011). Psychological Assessment, 23, 1034–1040.

European Journal of Psychological Assessment (2018), 34(1), 6–13

S. Sauer et al., Mindful Machine Learning

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning. Elements (Vol. 1), New York City, NY: Springer.

Hoefling, V., Moosbrugger, H., Schermelleh-Engel, K., Heidenreich, T., & Höfling, V. (2011). Mindfulness or Mindlessness? European Journal of Psychological Assessment, 27, 59–64. doi: 10.1027/ 1015-5759/a000045 Hofmann, S. G., Sawyer, A. T., Witt, A. A., & Oh, D. (2010). The effect of mindfulness-based therapy on anxiety and depression: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 169–183.

Kohls, N., Sauer, S., & Walach, H. (2009). Facets of mindfulness – results of an online study investigating the Freiburg mindfulness inventory. Personality and Individual Differences, 46, 224–230.

Kuhn, M. (2008). Building predictive models in R using the Caret package. Journal of Statistical Software, 28, 1–26.

Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363–374. doi: 10.2307/2529786 Langer, E. J., & Moldoveanu, M. (2000). The construct of mindfulness. Journal of Social Issues, 56, 1–9.

Larsen, R. J., & Buss, D. M. (2008). Personality psychology.

New York City, NY: Mac Graw-Hill.

Leigh, J., Bowen, S., & Marlatt, G. A. (2005). Spirituality, mindfulness and substance abuse. Addictive Behaviors, 30, 1335–1341. doi: 10.1016/j.addbeh.2005.01.010 Mars, T. S., & Abbey, H. (2010). Mindfulness meditation practise as a healthcare intervention: A systematic review. International Journal of Osteopathic Medicine, 13, 56–66. doi: 10.1016/j. ijosm.2009.07.005 Sauer, S., Lemke, J., Wittmann, M., Kohls, N., Mochty, U., & Walach, H. (2012). How long is now for mindfulness meditators?

Personality and Individual Differences, 52, 750–754.

Sauer, S., Walach, H., & Kohls, N. (2010). Gray’s Behavioural Inhibition System as a mediator of mindfulness towards wellbeing. Personality and Individual Differences, 50, 506–551.

Sauer, S., Walach, H., Offenbächer, M., Lynch, S., & Kohls, N. (2011). Measuring mindfulness: A Rasch analysis of the Freiburg mindfulness inventory. Religions, 2, 693–706.

Sauer, S., Walach, H., Schmidt, S., Hinterberger, T., Lynch, S., Büssing, A., & Kohls, N. (2013). Assessment of mindfulness: review on state of the art. Mindfulness, 4, 3–17.

Sauer, S., Ziegler, M., Danay, E., Ives, J., & Kohls, N. (2013).

Specific objectivity of mindfulness – a Rasch analysis of the Freiburg Mindfulness Inventory. Mindfulness, 4, 45–54. doi: 10.1007/s12671-012-0145-y Sedlmeier, P., Eberth, J., Schwarz, M., Zimmermann, D., Haarig, F., Jaeger, S., & Kunze, S. (2012). The psychological effects of meditation: A meta-analysis. Psychological Bulletin, 138, 1139–1171. doi: 10.1037/a0028168 Shipp, M. A., Ross, K. N., Tamayo, P., Weng, A. P., Kutok, J. L., Aguiar, R. C. T., . . . Golub, T. R. (2002). Diffuse large B-cell lymphoma outcome prediction by gene-expression profiling and supervised machine learning. Nature Medicine, 8, 68–74. doi: 10.1038/nm0102-68 Tang, Y.-Y., & Posner, M. I. (2013). Tools of the trade: Theory and method in mindfulness neuroscience. Social Cognitive and Affective Neuroscience, 8, 118–120. doi: 10.1093/scan/ nss112 Teper, R., & Inzlicht, M. (2013). Meditation, mindfulness and executive control: The importance of emotional acceptance and brain-based performance monitoring. Social Cognitive and Affective Neuroscience, 8, 85–92. doi: 10.1093/scan/nss045

Ó 2016 Hogrefe Publishing

S. Sauer et al., Mindful Machine Learning

Van Dam, N. T., Earleywine, M., & Borders, A. (2010). Measuring mindfulness? An Item Response Theory analysis of the Mindful Attention Awareness Scale. Personality and Individual Differences, 49, 805–810. doi: 10.1016/j.paid.2010.07.020 Ye, Q.-H., Qin, L.-X., Forgues, M., He, P., Kim, J. W., Peng, A. C., . . .

Wang, X. W. (2003). Predicting hepatitis B virus-positive metastatic hepatocellular carcinomas using gene expression profiling and supervised machine learning. Nature Medicine, 9, 416–423. doi: 10.1038/nm843 Zainal, N. Z., Booth, S., & Huppert, F. A. (2013). The efficacy of mindfulness-based stress reduction on mental health of breast cancer patients: A meta-analysis. Psycho-Oncology, 22, 1457–1465. doi: 10.1002/pon.3171

13

Sebastian Sauer FOM University of Applied Sciences Arnulfstr. 30 80335 Munich Germany Tel. +49 89 202 452-0 Fax +49 89 202 452-28 E-mail Sebastian.sauer@fom.de

Received November 19, 2014 Revised received May 31, 2015 Accepted July 6, 2015 Published online April 22, 2016

Ó 2016 Hogrefe Publishing

European Journal of Psychological Assessment (2018), 34(1), 6–13

Original Article

Construct Validity of the Eyberg Child Behavior Inventory (ECBI) An Exploratory Structural Equation Modeling Approach Silje Sommer Hukkelberg,1 Charlotte Reedtz,2 and John Kjøbli1 1

Norwegian Center for Child Behavioral Development, University of Oslo, Norway

2

Faculty of Health Science, The Arctic University of Norway

Abstract: Despite the wide use of the Eyberg Child Behavior Inventory (ECBI), its construct validity is still an issue of debate. Whereas previous studies primarily have relied on confirmatory factor analyses (CFAs), the present study aimed to evaluate the Norwegian 22-item version of the ECBI intensity scale using exploratory structural equation modeling (ESEM), which offers more flexibility compared to traditional CFA. The study used data from three samples of parents with children with emerging or existing problem behaviors. Results showed that the ESEM model showed good fit and identified the three factors Oppositional defiant behavior, Conduct problem behavior, and Inattentive behavior. In addition, the ESEM approach revealed several small and significant (λ .21) cross-loadings. The results illustrated the strengths of ESEM over CFA and suggest that ESEM provide a useful statistical framework to evaluate the structure of ECBI items.

Keywords: problem behavior, construct validity, CFA, exploratory structural equation modeling

Persistent problem behaviors in childhood and adolescence relate to poor health outcomes and negative long-term consequences later in life (e.g., Moffitt, 2006). In order to identify and reduce problem behaviors at an early age, psychometrically sound and accurate measurement instruments are warranted. Parents are usually considered the most important informants when it comes to reports of youth problem behaviors (Myers & Winters, 2002).

The Eyberg Child Behavior Inventory (ECBI; Eyberg & Ross, 1978) is a commonly used rating scale used in both clinical work and research (Burns & Patterson, 2000) that relies on parents’ reports. In Scandinavia, the instrument has been standardized and validated in both Sweden (Axberg, Hanse, & Broberg, 2008) and in Norway (Reedtz et al., 2008). The instrument is widely used for screening purposes when providing interventions for children with emerging or existing conduct problems (Solholm, Kjøbli, & Christiansen, 2013), and it is used in several randomized trials examining the effectiveness of parent training interventions aimed at reducing conduct problems (e.g., Kjøbli, Hukkelberg, & Ogden, 2013; Reedtz, Handegård, & Morch, 2010).

Specifically, the ECBI comprises of items assessing externalizing problem behaviors in children age 2–16 years. The original instrument consists of a list of 36 behaviors, constituting two scales: (i) the Intensity Scale, which indicates how often problem behaviors occur, and (ii) the Problem European Journal of Psychological Assessment (2018), 34(1), 14–21 DOI: 10.1027/1015-5759/a000316

Scale, identifying which specific behaviors are considered as a problem by the parents (Robinson, Eyberg, & Ross, 1980). The ECBI has shown good psychometric properties (Burns & Patterson, 1991, 2000) and high stability over time, both in children (Robinson, Eyberg, & Ross, 1980) and in adolescents (Eyberg & Robinson, 1983). Adequate test-retest, split-half, and internal consistency reliabilities have also been reported (Burns & Patterson, 1991; Eyberg, 1992). Nevertheless, a study of two large samples of children and adolescents (of ages 2 and 17 years) on the original 36-item version, using exploratory and confirmatory factor analyses (Burns & Patterson, 2000), revealed a four-factor solution consisting of three meaningful factors, in addition to one poorly defined factor reflecting response bias. The three meaningful factors, denoted Oppositional defiant behavior, Inattentive behavior, and Conduct problem behavior, encompassed 22 of the 36 items, suggesting a shorter version of the ECBI. Whereas the Oppositional defiant behavior factor includes aversive, yet nondestructive behaviors, directed toward adults, such as arguing and tantrums, the Inattentive behavior factor consists of items associated with inattention, poor concentration, and distractibility, similar to ADHD (Burns & Patterson, 2000). The third factor, Conduct problem behavior, reflects overt and covert conduct problem behavior, and actions involving aggression, destruction of objects, lying, and stealing. Ó 2016 Hogrefe Publishing

S. S. Hukkelberg et al., Construct Validity of the Eyberg Child Behavior Inventory (ECBI)

However, research during the past decade has identified different constellations of the ECBI items. According to factor analytic studies, the ECBI items might be considered as unidimensional (Burns & Patterson, 1990, 2000; Burns et al., 1991; Eyberg & Robinson, 1983; Robinson et al., 1980; Weis, Lovejoy, & Lundahl, 2005) or multidimensional (Axberg, Hanse, & Broberg, 2008; Burns & Patterson, 2000; Burns et al., 1991; Butler, 2010; Weis, Lovejoy, & Lundahl, 2005; Werba, 2003). However, many of these studies relied on traditional confirmatory factor analysis (CFA). That is, items expected to load on a construct are freely estimated, whereas others are fixed to zero.

Consequently, possible cross-loadings are assumed to be zero, which may undermine how the items relate to the different latent constructs. If, in fact, one or more items have substantial cross-loadings on another latent factor, a CFA approach is associated with failure to identify behaviors with weak discriminant validity and produce biased results in terms of inflated factor correlations and loadings (Asparouhov & Muthén, 2009). Given that latent problem behavior constructs are closely related, it seems unlikely that cross-loadings in the measurement model do not exist.

Methods that are more sophisticated have been developed that allow more factor analytic modeling flexibility, including cross-loadings. The exploratory structural equation modeling (ESEM) approach is an integration of CFA and exploratory factor analysis (EFA). It includes EFA measurement parts instead of the traditional CFA measurement model parts, but the statistical advantages of CFA (Marsh, Morin, Parker, & Kaur, 2014). That is, cross-loadings are allowed and the factor loading matrices can be rotated (see Asparouhov & Muthén, 2009 for more details).

The aims of the present study were to investigate the construct validity of the 22-item ECBI intensity scale (Burns & Patterson, 2000) by applying the ESEM approach, and to compare the CFA and ESEM procedures. We are not aware of any study evaluating the ECBI intensity scale using ESEM. Thus, this application of ESEM may contribute to a better understanding of the ECBI 22-item version.

Method Participants Data were derived from participants enrolled for three randomized trials of interventions aimed at reducing and preventing problem behaviors (Kjøbli, Hukkelberg, & Ogden, 2013; Kjøbli & Ogden, 2012, 2014). The families in the studies were recruited from all five health regions in Norway. The interventions were different in intensity and Ó 2016 Hogrefe Publishing

15

scope, and therefore aimed toward different target groups.

In line with standard procedures in the children’s services, the interventions were offered based on practitioners’ clinical judgments rather than formal screening. In all samples, children (and parents) were excluded from participation if they were diagnosed with autism, had been exposed to documented sexual assaults, were intellectually disabled, or had parents with serious mental health problems. The Regional Committee for Medical Research Ethics approved all three studies.

Sample 1 A total of 137 families from 11 agencies situated in different municipalities in Norway participated in the study (Kjøbli, Hukkelberg, & Ogden, 2013). No child was excluded based on the exclusion criteria. Children were between the ages of 2 and 12 years (M = 8.56, SD = 2.35) and 50 (36.5%) were girls. The mean score on the ECBI intensity scale was 83.09 (SD = 21.10), and the item mean was 3.78 (SD = 0.96). The average age of the reporting parent was 37.42 years (SD = 6.34). Among the participating children, 66 (48.2%) lived with both biological parents, 21 (15.3%) with parents who were married or cohabiting with another adult, and 50 (36.5%) lived with single parents (divorced, separated, or never married). The average gross annual family income was 509,609 Norwegian Kroner (SD = 347,701), which is approximately $83,542 and represents a middle income level (Statistics Norway, 2012). According to parent selfreport, 37 (27%) had a college or higher university degree, 83 (60.6%) had finished high school, and 17 (12.4%) had only completed junior high school or elementary school.

Most of the parents had Norwegian background (126 or 92%), only one (0.7%) was from another western European country, and 8 (7.3%) reported “other” ethnicity.

Sample 2 A total of 201 families from 9 municipalities in Norway participated in the study (Kjøbli & Ogden, 2014). The mean score on the ECBI intensity scale was 75.24 (SD = 19.24), and the item mean was 3.42 (SD = 0.87). One child was excluded because of a diagnosis of autism. The 198 children in this study ranged from 3 to 12 years of age at intake (M = 7.64, SD = 2.19), and 39 (19.7%) were girls. The average age of the reporting parent was 36.30 years (SD = 6.07).

Among the participating children, 106 (53.5%) lived with both biological parents, 29 (14.6%) with parents who were married or cohabiting with another adult, and 63 (31.8%) lived with single parents (divorced, separated, or never married). The average family income was 564,088 Norwegian Kroner (SD = 267,049), which is approximately $96,756 and represents an upper middle income level (Statistics Norway, 2012). According to parent self-report, 80 (40%) had a college or higher university degree,

European Journal of Psychological Assessment (2018), 34(1), 14–21

16

S. S. Hukkelberg et al., Construct Validity of the Eyberg Child Behavior Inventory (ECBI)

96 (48.5%) had finished high school, and 22 (11.1%) had completed only junior high school or elementary school.

Most of the parents reported to have Norwegian background (182 or 92%), two (1%) were from another western European country, and 14 (7.1%) reported “other” ethnicity.

Sample 3 A total of 216 families from 9 municipalities in Norway participated in the study (Kjøbli & Ogden, 2012). Children were recruited from schools and kindergartens. One child was excluded because of a diagnosis of autism. The children in this study ranged from 3 to 12 years of age at intake (M = 7.28, SD = 2.61), and 69 (31.9%) were girls. The mean score on the ECBI intensity scale was 76.05 (SD = 18.47), and the item mean was 3.46 (SD = 0.84). The average age of the parents was 35.31 years (SD = 6.08). One hundred ten children (50.9%) lived with both biological parents, 27 (12.5%) with parents who were married or cohabiting with another adult, and 79 (36.6%) lived with single parents (divorced, separated, or never married).

The average gross annual family income was 539,107 Norwegian Kroner (SD = 328,293), which is approximately $88,815 and represents an upper middle income level (Statistics Norway, 2012). According to self-reports, 85 of the parents (39.4%) had a college or higher university degree, 114 (52.8%) had finished high school, and 17 (7.9%) had completed only junior high school or elementary school. Most of the parents had Norwegian background (202 or 93.5%), four (1.9%) were from other western European countries, and the remaining ten (4.6%) reported “other” ethnicity.

Procedures The eligible families were informed of the studies and agreed to participate by signing a written informed consent document. In all samples, the families were assessed before (pretest), immediately after (posttest), and half a year after (follow-up) the completion of the interventions. Only the pretest data were used in the present study. The assessments took place at the agencies where the interventions were offered and lasted for about 1 hr. The assessment sessions were administered by research staff, employed and trained for the purposes of collecting data for the studies.

If parents needed directions when completing the questionnaires, the research staff would provide help in a similar manner across cases. The parents completed questionnaires about family demographics, parenting practices, and child behavior. For budgetary and logistic reasons, only the parent who reported having spent the most time with the child was asked to complete the questionnaires.

European Journal of Psychological Assessment (2018), 34(1), 14–21

The parents were not offered any economic compensation for participating in the study.

Measurement Problem behaviors were assessed using the 22-item ECBI scale (Burns & Patterson, 2000). The ECBI consists of an intensity scale that measures how often each of the 22 problem behaviors occurs, and one dichotomous problem scale that identifies whether or not each behavior is considered a problem for the parent. The present study deals with the intensity scale only. A 7-point Likert scale is used to rate how often each of the behaviors occurs (1 = Never, 7 = Always). Based on the number of response options, items were treated as continuous. The ECBI scale is a self-administered parent report and takes about 10 min to complete. It was adapted to Norway by Professor Willy-Tore Mørch and his colleagues at The Arctic University of Norway, and translated from English to Norwegian through the process of translation and back-translation in a team with advanced English skills. In Norway, the ECBI has been standardized in a sample of children aged 4–12 years (N = 4,063, mean = 89.9 on the intensity scale; Reedtz et al., 2008).

Statistical Analyses All factor models were investigated using Mplus 6.0 (Muthén & Muthén, 1998–2011) with robust estimation techniques (maximum likelihood estimation with robust standard errors, MLR; Muthén & Muthén, 2008). Assessment of overall model fit was based on multiple fit indices and included w2, RMSEA (Root Mean Square Error of Approximation), TLI (Tucker-Lewis Reliability Index), and CFI (Comparative Fit Index). Guidelines suggest that CFI and TLI values greater than .90 and .95 are considered to represent good and acceptable model fit, respectively (Bentler, 1990). Furthermore, Browne and Cudeck (1993) advocate that values of RMSEA less than .05 reflect good fit, and values as high as .08 represent acceptable fit.

Residual covariances were included in models based on modification indices (MI) when this seemed to be reasonable, for example, item similarities in content or wording.

Missing data were estimated using full information maximum likelihood estimation that assumes that missing data points have an expectation equal to a model-derived value estimated from the remaining data points (Muthén & Muthén, 2006). The ESEM model was specified with oblique Geomin rotation, as this is advantageous when suspecting variable complexity greater than 1 (Brown, 2006).

Ó 2016 Hogrefe Publishing

S. S. Hukkelberg et al., Construct Validity of the Eyberg Child Behavior Inventory (ECBI)

17

Table 1. Means, standard deviations, skewness, and kurtosis for the 22 problem behavior items across samples Sample 1 (N = 137) M 5. Refuses to do chores when asked 7. Refuses to go to bed on time 8. Does not obey house rules on his/her own 9. Refuses to obey until threatened with punishment 10. Acts defiant when told to do something 11. Argues with parents about rules 12. Gets angry when doesn’t get his/her own way 13. Has temper tantrums 14. Sasses adults 17. Yells or screams 19. Destroys toys and other objects 21. Steals 22. Lies 23. Teases or provokes other children 24. Verbally fights with friends his/her own age 25. Verbally fights with sisters and brothers 26. Physically fights with friends his/her own age 27. Physically fights with sisters and brothers 30. Is easily distracted 31. Has short attention span 32. Fails to finish tasks or projects 34. Has difficulty concentrating on one thing

4.34 3.96 4.45 4.11 4.29 4.34 3.76 4.13 3.53 3.92 2.52 1.66 3.02 3.54 3.36 4.45 2.20 3.02 4.93 4.74 4.40 4.09

SD

Skew

1.46 1.74 1.38 1.66 1.45 1.67 1.57 1.72 1.78 1.70 1.57 1.19 1.62 1.67 1.51 1.77 1.29 1.96 1.71 1.75 1.74 1.79

Kurt

0.47 0.07 0.30 0.38 0.25 0.23 0.52 0.30 0.10 0.29 0.77 2.17 0.44 0.06 0.22 0.70 1.22 0.52 0.68 0.40 0.27 0.07

0.32 1.12 0.20 0.99 0.75 0.89 0.68 0.89 1.14 1.05 0.48 4.32 0.97 1.17 0.89 0.56 0.99 1.14 0.55 0.91 0.93 1.11

Sample 2 (N = 201) M 3.75 2.94 3.76 3.57 3.74 3.71 4.35 3.56 3.17 3.32 2.45 1.57 3.06 3.48 3.65 3.81 2.41 2.52 4.58 4.35 3.84 3.67

SD 1.34 1.60 1.44 1.53 1.30 1.60 1.48 1.50 1.54 1.59 1.51 1.03 1.39 1.37 1.30 1.84 1.41 1.69 1.56 1.64 1.62 1.60

Skew 0.21 0.84 0.14 0.01 0.03 0.38 0.10 0.02 0.28 0.25 1.10 2.29 0.59 0.30 0.21 0.20 0.94 0.95 0.22 0.13 0.16 0.20

Kurt 0.28 0.05 0.66 0.87 0.57 0.66 0.87 0.93 0.90 0.97 0.53 5.95 0.32 0.46 0.50 1.01 0.15 0.06 0.82 0.95 0.90 0.91

Sample 3 (N = 216) M 4.04 3.58 3.87 3.92 4.25 4.05 4.83 3.96 3.00 3.71 2.44 1.51 2.63 3.09 3.26 3.96 1.97 2.67 4.24 3.91 3.72 3.46

SD 1.40 1.72 1.52 1.50 1.33 1.68 1.38 1.50 1.52 1.58 1.48 1.01 1.54 1.56 1.36 1.98 1.21 1.77 1.66 1.67 1.53 1.70

Skew 0.30 0.28 0.14 0.23 0.20 0.10 0.50 0.20 0.34 0.09 0.97 2.24 0.81 0.29 0.30 0.32 1.43 0.78 0.15 0.12 0.20 0.46

Kurt 0.66 0.95 0.82 0.71 0.62 0.92 0.23 0.64 0.86 0.80 0.08 40.76 0.12 10.07 0.67 1.24 1.75 0.56 0.96 1.04 0.82 0.95

Notes. N = number of children in each sample, M = mean, SD = Standard Deviation, Skew = skewness, Kurt = kurtosis. Total mean for sample 1 was 82.76 (SD = 35.70), sample 2: M = 75.26 (SD = 32.88), and sample 3 M = 76.07 (SD = 33.60). Item numbers are similar to the original ECBI 36-item version.

Results Descriptives Table 1 provides descriptive statistics for the 22 items in the three independent samples. As shown, most items displayed mean values around the midpoint, except from item 21 (“Steals”) which showed a low mean value, resulting in a leptokurtic and skewed distribution across the samples (skewness ranged from 2.17 to 2.29, kurtosis ranged from 5.95 to 4.32). However, inspections of correlations revealed that this item, in particular, correlated positively with age in all three samples, suggesting that the low mean value reflects that it may be more relevant for older children.

The respondents of the first sample reported a higher mean value across items (M = 82.76, SD = 35.7) compared to sample 2 (M = 75.26, SD = 32.88) and sample 3 (M = 76.07, SD = 33.60). This was expected since these participants were enrolled for the most comprehensive intervention.

The total mean score of the 22-items across all three samples (N = 554) was 77.32 (SD = 19.78; min = 22, max = 140, range = 22–154).

Exploratory Structural Equation Modeling Table 2 shows the primary and secondary factor loadings of the items on the ECBI. As shown, the items fitted a threefactor solution with the factors F1 reflecting Oppositional Ó 2016 Hogrefe Publishing

defiant behavior (α = .89), F2 reflecting Conduct problem behavior (α = .76), and F3 reflecting Inattentive behavior (α = .92). In addition, several small cross-loadings (λ .21) were present. The ESEM model provided a reasonable solution in terms of fit indices, w2 = 468.782 (163), RMSEA = .058 (.052–.064), CFI = .94, and TLI = .92. Noteworthy, item 19 (“Destroys toys and other objects”) showed poor discriminant validity in terms of a week loading (λ .29) on both the first and the second factor. Except from this item, indicators clearly clustered into three main factors with the majority of items loading λ .40 for the first factor, and λ .82 for the third factor. The items on the second factor showed larger dispersion; loadings ranged from λ .28 to λ .82, that is, 5 out of 8 items showed less than 10% overlap in variance with the factor.

Tabachnick and Fidell (2001) suggest a minimum factor loading of .32. The results show that only one cross-loading was above this value, that is, item 14 (“Sasses adults”) which loaded .40 on the Oppositional defiant behavior factor, and .36 on the third Inattentive behavior factor.

The correlations between latent factors ranged from r = .35–.45 (Figure 1).

The model included residual covariances between item 17 (“Yells or screams”), item pair 12 (“Gets angry when doesn’t get his/her own way”) and 13 (“Has temper tantrums”), between the item pair 21 (“Steal”) and 22 (“Lies”), and between item pair 25 (“Verbally fights with sisters and brothers”) and 27 (“Physically fights with sisters and European Journal of Psychological Assessment (2018), 34(1), 14–21

18

S. S. Hukkelberg et al., Construct Validity of the Eyberg Child Behavior Inventory (ECBI)

Table 2. Standardized factor loadings for problem behavior items (standard errors) using an ESEM approach F1 F1: Oppositional defiant behavior 5. Refuses to do chores when asked 7. Refuses to go to bed on time 8. Does not obey house rules on his/her own 9. Refuses to obey until threatened with punishment 10. Acts defiant when told to do something 11. Argues with parents about rules 12. Gets angry when doesn’t get his/her own way 13. Has temper tantrums 14. Sasses adults 17. Yells or screams F2: Conduct problem behavior 19. Destroys toys and other objects 21. Steals

0.66 0.51 0.40 0.64 0.78 0.65 0.85 0.76 0.40 0.69

(0.04) (0.04) (0.05) (0.04) (0.04) (0.04) (0.03) (0.03) (0.04) (0.04)

0.23 (0.05) 0.07 (0.05)ns

22. Lies 23. Teases or provokes other children 24. Verbally fights with friends his/her own age 25. Verbally fights with sisters and brothers 26. Physically fights with friends his/her own age 27. Physically fights with sisters and brothers F3: Inattentive behavior 30. Is easily distracted 31. Has short attention span 32. Fails to finish tasks or projects 34. Has difficulty concentrating on one thing

F2 0.08 0.07 0.24 0.01 0.03 0.10 0.03 0.06 0.01 0.07

F3

(0.05)ns (0.05)ns (0.04) (0.03)ns (0.04)ns (0.05) (0.04)ns (0.05)ns (0.02)ns (0.05)ns

0.06 0.02 0.13 0.06 0.05 0.02 0.05 0.00 0.36 0.09

(0.05)ns (0.05)ns (0.05) (0.05)ns (0.05)ns (0.04)ns (0.04)ns (0.03)ns (0.05) (0.05)ns

0.29 (0.05) 0.28 (0.05)

0.10 (0.05) 0.14 (0.05)

0.02 0.00 0.00 0.21 0.04 0.24

(0.05)ns (0.03)ns (0.03)ns (0.05) (0.05)ns (0.05)

0.82 0.68 0.28 0.64 0.30 0.29

(0.04) (0.04) (0.06) (0.05) (0.06) (0.05)

0.24 0.02 0.11 0.02 0.01 0.08

(0.05) (0.04)ns (0.04) (0.05)ns (0.03)ns (0.05)ns

0.01 0.08 0.02 0.02

(0.02)ns (0.03)ns (0.03)ns (0.03)ns

0.04 0.00 0.01 0.04

(0.03)ns (0.01)ns (0.03)ns (0.04)ns

0.84 0.96 0.85 0.82

(0.02) (0.02) (0.02) (0.02)

Note. N = 554. Loadings were significant at p < .05 when not indicated as ns. Significant loadings λ > 0.20 are marked in bold.

5 7 8 9 10 11 12 13 14 17

19 21 22 23 24 25 26

Finally, we performed an ESEM of the ECBI items, excluding the items with a factor loading less than .30 (item 19, 21, 24, and 27). All these items reflected the latent Conduct problem behavior, thus leaving the revised factor F2 with four indicators. In addition, this model included the residual covariances described above, related to items reflecting the Oppositional Defiant behavior factor. This ESEM model resulted in a substantial increase in fit (w2 = 252.62 (99), RMSEA = .053 (.045–.061), CFI = .96, and TLI = .94), and a model with factor loadings above .30. Nevertheless, a substantial cross-loading for item 14 was still present (λ = .35), in addition to several small but significant cross-loadings (λ < .23).

27

Discussion

Figure 1. ESEM model showing primary factor loadings and crossloadings (λ 0.32). All correlations between latent factors were significant (p < .05).

brothers”). Residual covariances between these items were allowed based on substantial overlap in content or wording, in addition to a large MI (MI 38.99).

European Journal of Psychological Assessment (2018), 34(1), 14–21

The purpose of this study was to examine the discriminant validity of the 22-item ECBI intensity scale (Burns & Patterson, 2000), in a large sample of three independent samples of children with emerging or existing conduct problems. The instrument assesses externalizing problem behaviors in children aged 2–16 years. Whereas previous factor analytic work has produced both one-dimensional and multidimensional structures using CFA, no studies have considered the ECBI using the more recent ESEM approach (Asparouhov & Muthén, 2009; Marsh et al., Ó 2016 Hogrefe Publishing

S. S. Hukkelberg et al., Construct Validity of the Eyberg Child Behavior Inventory (ECBI)

2009). In this study, we examined ECBI using ESEM, which offers more flexibility than the traditional CFA, to get a better understanding of the underlying latent structure of ECBI. ESEM differs from traditional CFA in that the latter does not allow all factor loadings to be freely estimated (i.e., cross-loadings), which may result in, for example, poor model fit and overestimated factor correlations. The ESEM model was specified with three latent factors, and oblique Geomin-rotated since cross-loadings were expected in the measurement model (Brown, 2006). The findings from this study showed that the ESEM approach was able to produce a reasonable good fitting model, comprising of the three factors Oppositional defiant behavior, Conduct problem behavior, and Inattentive behavior. This factor structure has previously been evidenced in the literature (e.g., Axberg, Hanse, & Broberg, 2008; Burns & Patterson, 2000; Weis, Lovejoy, & Lundahl, 2005). However, as expected, the ESEM results revealed several small (λ .21) and significant cross-loadings for five of the ECBI items.

Noteworthy, several items, especially on the Conduct problem behavior factor, showed low factor loadings (λ .30). In consequence, these items share less than 9% of the variance with the factor. This may be due to the complex nature of the conduct problem items, containing both overt and covert behavioral descriptions. Two of the items showing low factor loadings were related to brother/sister interaction and therefore only relevant for those children with siblings. Furthermore, some items may be more relevant to older children and therefore generate low factor loadings. Thus, it may be that these items would show higher factor loadings in a sample comprising of older children. The relative high items loadings in Oppositional Defiant Behavior and Inattentive Behavior point toward these factors as more stable constructs compared to the Conduct Problem behavior factor. Furthermore, only one item showed a cross-loading higher than the recommended cut-off loading of .32 – item 14 (“Sasses adults) loaded on both Oppositional defiant and Inattentive behavior. This cross-loading may be explained by parents/adults being the ones supervising the tasks measured by Inattentive behaviors, but also the ones who must deal with oppositional defiant behavior. All in all, the findings illuminate the strength of the ESEM, and the need to include crossloadings when measuring factors of problem behaviors.

Nevertheless, it may be worthwhile to consider the items’ relevance and conceptual importance in the overall evaluation of problem behaviors, especially the ones reflecting Conduct Problem behavior. The result of the revised ECBI model suggests that removing those items with a low factor loading (λ < .30) would result in a better fitting model. The applicability of such a short measure of oppositional defiant behavior should, however, be investigated further. Ó 2016 Hogrefe Publishing

19

Several limitations of this study should be mentioned.

First, the findings from this study do not necessarily generalize to the normative population, as samples included children with emerging or existing conduct problems. The children varied considerably in their range of behavioral problems, from a minimum level to considerable problem behaviors. From practice, we know that conduct problems may not always be the cause when parents search for help, for example, a high conflict level in the family may be the main problem. The average child in this study did, nevertheless, display problem behaviors well above the Norwegian norm level. Second, as the aim of this study was to consider the construct validity of the 22-item version of the ECBI, we did not include analyses on the original 36-item version. It seems much needed to test whether the dimensionality from Burns and Patterson’s study (2000) could be replicated in Scandinavia. Third, the cutoff values of RMSEA, CFI, and TLI are developed based on simulated data. There exists little empirical work on how these values apply for ESEM.

Implications for Practice Prevention of behavior problems in children has been a prioritized task throughout Scandinavia the last decades, and hence the availability of sound and accurate measurement is crucial. For different practical reasons the ECBI has been preferred by Scandinavian researchers; the instrument is easily accessible, short, the response rate has typically been very good from parents, and it does not demand sertification or training to use and score it. The ECBI has been standardized and validated in Scandinavia (Axberg, Hanse, & Broberg, 2008; Reedtz et al., 2008), and has because of its good psychometrical properties been widely used for screening purposes in clinical practice (Solholm, Kjøbli, & Christiansen, 2013), as well as in research evaluating interventions for children at risk for developing conduct problems (e.g., Kjøbli, Hukkelberg, & Ogden, 2013, Reedtz, Handegård, & Morch, 2010). At the time being, it is the only widely available scale that has been translated into the Scandinavian languages, along with the Child Behavior Checklist (CBCL; Achenbach, 1991), and Strengths and Difficulties Questionnaire (SDQ; Goodman, 2001).

In this study, we identified a multidimensional structure as the best representation of the Norwegian version of the ECBI. The factors Oppositional defiant behavior, Conduct problem behavior, and Inattentive behavior have been supported by several studies and seem to represent a stable constellation of the ECBI items. Hence, results provide support that practitioners may use these factors to identify and assess children and adolescent with problem behaviors.

However, the multidimensional factor structure has consequences for its validity, reliability, and use (Furr, 2011).

European Journal of Psychological Assessment (2018), 34(1), 14–21

20

S. S. Hukkelberg et al., Construct Validity of the Eyberg Child Behavior Inventory (ECBI)

Its validity is affected since a correct interpretation depends on the match between the scale’s intended constructs and the constructs that were revealed. Also, the latent structure affects the internal consistency between items. Following from the validity and reliability of the ECBI, the factor structure is relevant in terms of understanding the psychological phenomenon of problem behaviors as being assessed by the ECBI scale’s items. Furthermore, practitioners should be aware that some items may be more appropriate at older age groups (e.g., “Steals”) or children with siblings. In young children, there may be less differentiation among externalizing behavior problems (Cole, Truglio, & Peeke, 1997). Also, several item pairs are naturally connected, that is, a high score on one item also includes a high score on connected items. Overall, findings suggest that practitioners should examine scores on the factors instead of relying on a global problem behavior score.

A step in the future would be to adopt scales measuring behavior problems in children in closer accordance with the criteria for oppositional defiant disorder (ODD) and conduct disorder (CD) diagnosis in the DSM-5/ICD systems.

However, thus far, the ECBI has proven to be a feasible, valid, and reliable measure of child problem behaviors in Scandinavia.

References Achenbach, T. M. (1991). Manual for the child behavior checklist/ 4–18 and 1991 profile. Burlington, VT: Department of Psychiatry, University of Vermont.

Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397–438.

Axberg, U., Hanse, J. J., & Broberg, A. (2008). Parents’ description of conduct problems in their children: A test of the Eyberg Child Behavior Inventory (ECBI) in a Swedish sample aged 3–10.

Scandinavian Journal of Psychology, 49, 497–505. doi: 10.1111/ j.1467-9450.2008.00670.x Bentler, P. M. (1990). Comparative Fit Indexes in Structural Models. Psychological Bulletin, 107, 238–246.

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage.

Burns, G. L., & Patterson, D. R. (1990). Conduct problem behaviors in a stratified random sample of children and adolescents: New standardization data on the Eyberg Child Behavior Inventory.

Psychological Assessment: A Journal of Consulting and Clinical Psychology, 2(4), 391.

Burns, G. L., & Patterson, D. R. (1991). Factor structure of the Eyberg Child Behavior Inventory: Unidimensional or multidimensional measure of disruptive behavior? Journal of Clinical Child Psychology, 20, 439–444.

Burns, G. L., & Patterson, D. R. (2000). Factor structure of the Eyberg Child Behavior Inventory: A parent rating scale of oppositional defiant behavior toward adults, inattentive behavior, and conduct problem behavior. Journal of Clinical Child Psychology, 29, 569–577. doi: 10.1207/S15374424JCCP2904_9

European Journal of Psychological Assessment (2018), 34(1), 14–21

Burns, G., Patterson, D. R., Nussbaum, B. R., & Parker, C. M. (1991). Disruptive behaviors in an outpatient pediatric population: Additional standardization data on the Eyberg Child Behavior Inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 3, 202–207. doi: 10.1037/ 1040-3590.3.2.202 Butler, A. M. (2010). Cross-racial measurement equivalence of the Eyberg Child Behavior Inventory Factors among young African American and European American children. Dissertation Abstracts International: Section B: The Sciences and Engineering, 70, 7844.

Cole, D. A., Truglio, R., & Peeke, L. (1997). Relation between symptoms of anxiety and depression in children: A MultitraitMultimethod-Multigroup Assessment. Journal of Consulting and Clinical Psychology, 65, 110–119.

Eyberg, S. M. (1992). Assessing therapy outcome with preschoolchildren – Progress and problems. Journal of Clinical Child Psychology, 21, 306–311.

Eyberg, S. M., & Robinson, E. A. (1983). Conduct problem behavior: Standardization of a behavioural rating scale with adolescents. Journal of Clinical Child Psychology, 12, 347–354.

Eyberg, S. M., & Ross, A. W. (1978). Assessment of child-behavior problems – Validation of a New Inventory. Journal of Clinical Child Psychology, 7, 113–116.

Furr, M. (2011). Scale Construction and psychometrics for social and personality psychology. London, UK: The SAGE Library of Methods in Social and Personality Psychology.

Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Journal of the American Academy of Child & Adolescent Psychiatry, 11, 1337–1345. doi: 10.1097/ 00004583-200111000-00015 Kjøbli, J., Hukkelberg, S., & Ogden, T. (2013). A randomized trial of group parent training: Reducing child conduct problems in realworld settings. Behaviour Research and Therapy, 51, 113–121. doi: 10.1016/j.brat.2012.11.006 Kjøbli, J., & Ogden, T. (2012). A randomized effectiveness trial of brief parent training in primary care settings. Prevention Science, 13, 616–626. doi: 10.1007/s11121-012-0289-y Kjøbli, J., & Ogden, T. (2014). A randomized effectiveness trial of individual child social skills training: Six-month follow-up. Child and Adolescent Psychiatry and Mental Health, 8, 31. doi: 10.1186/s13034-014-0031-6 Marsh, H. W., Muthén, B., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A. J. S., & Trautwein, U. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students’ evaluations of university teaching. Structural Equation Modeling, 16, 439–476.

Marsh, H. W., Morin, A. J., Parker, P. D., & Kaur, G. (2014).

Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85–110. doi: 10.1146/annurev-clinpsy-032813-153700 Moffitt, T. E. (2006). Life-course-persistent versus adolescencelimited antisocial behavior. In D. Cicchetti & D. J. Cohen (Eds.), Developmental psychopathology: Risk, disorder and adaptation (2 ed., pp. 570–598). Mahwah, NJ: Wiley.

Muthén, B. O., & Muthén, L. K. (1998–2011). Mplus Software (Version 6.11). Los Angeles, CA: Muthén & Muthén.

Muthén, L. K., & Muthén, B. O. (2006). Mplus user's guide (4th ed.).

Los Angeles, CA: Muthén & Muthén.

Muthén, L. K., & Muthén, B. O. (2008). Mplus User’s Guide (5th ed.). Los Angeles, CA: Muthén & Muthén.

Myers, K., & Winters, N. C. (2002). Ten-year review of rating scales. I: overview of scale functioning, psychometric properties, and selection. Journal of the American Academy of Child & Adolescent Psychiatry, 41(2), 114–122.

Ó 2016 Hogrefe Publishing

S. S. Hukkelberg et al., Construct Validity of the Eyberg Child Behavior Inventory (ECBI)

Reedtz, C., Bertelsen, B., Lurie, J. I. M., Handegard, B. H., Clifford, G., & Morch, W. T. (2008). Eyberg Child Behavior Inventory (ECBI): Norwegian norms to identify conduct problems in children.

Scandinavian Journal of Psychology, 49, 31–38. doi: 10.1111/ j.1467-9450.2007.00621.x Reedtz, C., Handegård, B. H., & Morch, W. T. (2010). Promoting positive parenting practices in primary pare: Outcomes and mechanisms of change in a randomized controlled risk reduction trial. Scandinavian Journal of Psychology, 52, 131–137. doi: 10.1111/j.1467-9450.2010.00854.x Robinson, E. A., Eyberg, S. M., & Ross, A. (1980). The standardization of an inventory of child conduct problem behaviors.

Journal of Clinical Child Psychology, 9, 22–29. doi: 10.1080/ 15374418009532938 Solholm, R., Kjøbli, J., & Christiansen, T. (2013). Early initiatives for children at risk – Development of a program for the prevention and treatment of behavior problems in primary services.

Prevention Science, 14(6), 535–544.

Statistics Norway. (2012). Inntekt for par med barn og enslige forsørgere [Incomes for couples with children and single parents].

Retrieved from http://statbank.ssb.no/ statistikkbanken Tabachnick, B. G., & Fidell, L. S. (2001). Using Multivariate Statistics. Boston, MA: Allyn and Bacon.

Ó 2016 Hogrefe Publishing

21

Weis, R., Lovejoy, M., & Lundahl, B. W. (2005). Factor structure and discriminative validity of the Eyberg Child Behavior Inventory with young children. Journal of Psychopathology and Behavioral Assessment, 27, 269–278. doi: 10.1007/s10862005-2407-7 Werba, B. E. (2003). Standardization and cross-cultural validity of the Eyberg Child Behavior Inventory with Australian preschoolers. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63, 43–91.

Received February 6, 2015 Revision received July 9, 2015 Accepted July 11, 2015 Published online April 22, 2016 Silje Sommer Hukkelberg Department Atferdssenteret University of Oslo PO Box 7053 7053 Majorstuen 0306 Oslo Norway Tel. +47 92868553 E-mail s.s.hukkelberg@atferdssenteret.no

European Journal of Psychological Assessment (2018), 34(1), 14–21

Original Article

You Will Attain My Goal The Structure of Parental Goals for Children Based on an Adapted Version of the Aspirations Index Jelena Hollmann, Julia Gorges, and Elke Wild Department of Psychology, Bielefeld University, Germany

Abstract: This paper deals with parental goals for children, that is, goals parents want their children to attain by the time they reach adulthood. Presenting an adaption of the German Aspirations Index to assess parental goals for children (AI-PG), factorial structure and psychometric properties in a nonclinical sample of German parents of children between 10 and 15 years of age were investigated. Parents (NT1 = 948; NT2 = 670) rated the importance of different parental goals for children. The AI-PG structure was first examined in a subsample of mothers using confirmatory factor analyses. Main results provided support for the hypothesized hierarchical factorial structure. Accordingly, the given 21 items can be assigned to seven first-order factors that reflect goal-contents, and two second-order factors that reflect goals’ motivational orientation as either intrinsic or extrinsic. The questionnaire shows good psychometric properties (i.e., homogeneity, model-based reliability, and stability). Multiple-group analyses confirmed measurement invariance across informants (mothers and fathers), gender of target children (daughters and sons), and measurement time points. Small but significant relationships of the scales with need-threatened environment, parenting practices, and child internalizing behavior problems added to the scale’s construct validity. Results are discussed in terms of the fruitfulness of a concept of parental goals for children within self-determination theory.

Keywords: parental goals for children, Aspirations Index, hierarchical structure, reliability

Parents’ beliefs have a substantial influence on their children’s course of life (e.g., by communicating expectations and encouragement or by providing a learning environment; Sigel & McGillicuddy-DeLisi, 2002). Parents want their children to be happy and – in the long run – to be integrated into society (Deci & Ryan, 2000). Consequently, parents set goals for their children’s future. These parental goals for children refer to different outcomes and qualities of life that parents want their children to attain by the time they reach adulthood. According to self-determination theory (SDT; e.g., Deci & Ryan, 2000), parental goals for children can aim at a child’s personal growth (i.e., intrinsic goals) or direct a child’s behavior toward making a good impression on others (i.e., extrinsic goals). Promoting intrinsic parental goals for children is more beneficial for the child’s well-being and development because intrinsic goals focus on the satisfaction of the child’s inner needs and potential to grow as a person (Vansteenkiste, Niemiec, & Soenens, 2010).

One of the most commonly used standardized measures for personal goals for both parents and children is the Aspirations Index (AI; Kasser & Ryan, 1993, 1996). The AI measures the extent to which individuals strive for particular content-related goals that are either intrinsic or extrinsic.

Whereas many studies have dealt with personal goals, parental goals for children have received very limited attention (Sigel & McGillicuddy-DeLisi, 2002). In addition, European Journal of Psychological Assessment (2018), 34(1), 22–29 DOI: 10.1027/1015-5759/a000317

existing studies typically have focused on child-reported parental goals, although the parents themselves should know best which goals they have set for their children.

Therefore, the aim of the present paper is to present an adapted version of the AI that measures self-reported parental goals for children (AI-PG). Such a scale would enable researchers to gain a better understanding of the role of parental goals for children for the parent-child relationship, for example, regarding parental behavior and children’s developmental outcomes (Sigel & McGillicuddy-DeLisi, 2002).

According to SDT (cf. Deci & Ryan, 2000), individuals strive to satisfy inherent psychological needs for relatedness (i.e., having secure and caring social relationships), autonomy (i.e., being self-determined or self-caused instead of being controlled by an external agent), and competence (i.e., being effective and capable). Basic needs are considered as the motivational force behind intrinsic (versus extrinsic) goal-setting, and goal-attainment is assumed to satisfy these needs. The AI focuses on these goals comprising four intrinsic goal-contents – personal development, health, contribution to the betterment of society, and close relationships with friends and family (Kasser & Ryan, 1996). According to Vansteenkiste and Ryan (2013), individuals facing situations that obstruct the satisfaction of their needs pay a short-term cost in the form of ill-being.

If needs are chronically thwarted, individuals develop Ó 2016 Hogrefe Publishing

J. Hollmann et al., Aspirations Index for Parental Goals for Children

need-substitutes (i.e., strategies to accommodate the experience of need-frustration). Need-substitutes are defined through three extrinsic goal-contents of AI such as popularity, attractiveness, and wealth. Thus, extrinsic goals have external indicators of worth that protect a person from the threat of need-frustration (e.g., feeling of insecurity) by preserving a sense of fleeting need-satisfaction. Over the long run, however, these extrinsic goals interfere with genuine need-satisfaction and, therefore, people feel empty and disappointed (Vansteenkiste & Ryan, 2013).

The AI is a validated questionnaire comprising a list of personal goals assigned to seven subscales reflecting goalcontents, and two higher-order factors reflecting goalrelated motivational orientation in terms of either intrinsic or extrinsic (Klusmann, Trautwein, & Lüdtke, 2005). In statistical terms, the hierarchical structure of the AI reflects a second-order model. In such models, second-order factors have direct effects on the assigned first-order factors and indirect effects on the items. Accordingly, the importance of different goals is explained by different content factors (and residual), and different content factors are explained by different motivational orientations (and residual). Basically, this hierarchical structure was confirmed using both two-step principal component analysis (Klusmann et al., 2005) and multidimensional scaling analysis (Grouzet et al., 2005). However, health-related goals – assumed to be intrinsic – show cross-loadings on the extrinsic factor (Deci & Ryan, 1997). In fact, health-related goals (e.g., physical fitness) do not necessarily satisfy needs. Rather, these goals target making a good impression on others, which would be extrinsic.

To measure parental promotion of goals, Duriez, Soenens, and Vansteenkiste (2007) used the AI to assess the extent to which adolescents feel their parents have attached importance to different goals. They found that higher promotion of extrinsic goals negatively predicts parental need-supportive behavior. Kasser, Ryan, Zax, and Sameroff (1995) used the AI to assess parental goals for children. Mothers rated how important it was to them that their children attained certain goals. Results suggested that the AI worked reasonably well in that context. Unfortunately, as the primary research interest of Kasser et al.

(1995) was focused on children rather than parents, they did not report many details regarding psychometric qualities of the questionnaire.

Before researchers can understand the origin, effects, and transmission of parental goals for children, the construct itself should be described and analyzed. Benish-Weisman, Levy, and Knafo (2013) found that parents’ values for themselves and for their children are not identical. Grusec and Goodnow (1994) argued that parents recognize the difference between what is good for them and what may be good for their children. Malmberg, Ehrman, and Lithén Ó 2016 Hogrefe Publishing

23

(2005) found that adolescents’ and parents’ beliefs regarding future career, social and family life were similarly structured, but unrelated.

Parental goals for children can be defined as internal representations of desired states to be attained by one’s child in the future. There, goal-importance is an indicator of the extent to which the goal is endorsed, and the source (and orientation) of motivation for the goal can be either intrinsic or extrinsic. With regard to the inspection of different goal-contents of the AI, the change of perspective from intrapersonal to interpersonal implies only a discrepancy for goal-content related to community contribution. If an individual voluntarily helps others, the social responsibility is integrated into the self (Deci & Ryan, 2000). By contrast, parents who want their children help others may target external indicators of worth. Moreover, it is typically the case that children are uninterested in making a community contribution (Massey, Gebhardt, & Garnefski, 2008), therefore, parents may push their child to perform goal-directed behavior and neglect the child’s needs. Thus, we expect that community-contribution goals show a cross-loading on the extrinsic factor.

Parents who promote extrinsic more than intrinsic goals are more concerned with the impression their child makes, more often practice controlling parenting strategies, and are more likely to engage in stressful interpersonal comparisons and competitions (Duriez et al., 2007). The child – who is expected to perform goal-directed behavior to pursue the parental extrinsic goal – experiences introjected regulation of motivation (Deci & Ryan, 2000). Introjected regulation of motivation means that the child does what the parents want because, otherwise, the child would feel guilty about disappointing her/his parents’ expectations.

The present study investigated the structure and psychometric properties of an adapted and shortened version of the AI-PG, which has not been validated or published before. To validate the structure suggested by previous research and theoretical assumptions we compared the hypothesized higher-order factor model (Model-1) including cross-loadings on health (Deci & Ryan, 1997) and community contribution (caused by the change of perspective) against a model with a cross-loading on health only (Model-2a), against a model with a cross-loading on community contribution only (Model-2b), against a model without cross-loadings as initially hypothesized by AI authors (Model-2), and against a one higher-order factor model (general factor in Model-3; Duriez et al., 2007). All models are depicted in Figure 1. We used a subsample of mothers because our results would then be comparable to those from Kasser et al. (1995). Moreover, research has shown stronger effects of mother’s need-satisfaction on the children’s development, as compared to fathers’ (e.g., Niemiec et al., 2006). Next, we investigate the transferability of the European Journal of Psychological Assessment (2018), 34(1), 22–29

24

J. Hollmann et al., Aspirations Index for Parental Goals for Children

Figure 1. Alternative factors models of AI-PG. IG = intrinsic goals; EG = extrinsic goals; G = general factor; PG = personal growth; MR = meaningful relationship; CC = community contribution; H = health; W = wealth; F = fame; I = image.

hypothesized Model-1. The structure should be independent across different informants, child gender, and repeated measurements. We anticipated that the scale is reliable and stable because parental goals for children refer to long-term strivings. To add to the construct validity of the scale, we examined the relationships of relative European Journal of Psychological Assessment (2018), 34(1), 22â&#x20AC;&#x201C;29

importance of intrinsic and extrinsic goals with theoretically related variables. We expected to find highly extrinsic goals for children with single parents and parents with lower educational levels because of a higher likelihood of needfrustration in these groups. As a consequence of higher extrinsic parental goals for children, we expected negative Ă&#x201C; 2016 Hogrefe Publishing

J. Hollmann et al., Aspirations Index for Parental Goals for Children

relationships with autonomy-supportive and positive relationships with controlling parental practices. Finally, as a consequence of introjected regulation, we expected a positive association between the extrinsic goal promotion and internalizing child behavior problems.

Method Procedure Data stems from a 6-year longitudinal project supported by the German Federal Ministry of Education and Research (for more information, see, e.g., Chng, Wild, Hollmann, & Otterpohl, 2014). All data relevant for the present study were collected at the second (Grade 6) and third (Grade 7) measurement point (spring 2011 and spring 2012). Participants came from selected schools representing the highest track (N = 29) and the lowest track (N = 80) in two midsized towns in northern Germany, including neighboring villages. Eight highest track schools (with 35 classrooms) and 21 lowest track schools (at T1; at T2, this number fell to 19 schools due to two schools declining to participate for school-related reasons) participated in the study (for the lowest track schools, T1: 42 classrooms; T2: 39 classrooms). All children who had provided informed parental consent completed the questionnaires.

Due to the relatively greater school enrolments, the greater proportion of overall participants attended the highest school track (T1: 61%; T2: 62%). Parental participation was obtained via schoolchildren, who brought home a questionnaire. Parents were asked to complete the questionnaire and to send it back to the research team. The response rates for parents in the higher track schools were higher (T1: 77%; T2: 72%) than for lower track schools (T1: 54%; T2: 49%).

Participants A total of 948 parents at T1 (691 mothers, 103 fathers, 134 mothers and fathers together, and 20 other caretakers) and 670 parents at T2 (498 mothers, 71 fathers, 93 mothers and fathers together, and 3 other caretakers) were selected. The AI-PG was completed with respect to 511 boys (T2: 343) and 435 girls (T2: 323) aged between 10 and 14 years (M = 11.7, SD = 0.65, T2: M = 12.6; SD = 0.65). To ensure the comparability of participants across measurement points, only cases from parents who sent the questionnaire back at T1 and/or T2 were included in the analyses. The highest level of education held by at least one parent was classified according to the International Standard Classification of Education (HISCED; UNESCO, 1997): 450 had a preparatory Ó 2016 Hogrefe Publishing

25

college or university degree (high; T2: 377), 432 had intermediate general education (medium; T2: 267), and 58 had a lower education (low; T2: 23).

Measures Parental goals for children were assessed using adapted items from the German version of the AI (Klusmann et al., 2005). Based on preliminary item analyses and other research (e.g., Duriez et al., 2007), we selected three out of five items from each subscale that were deemed relevant regarding parental goals for children, and we rephrased them accordingly. The general introduction was rephrased to “What will your children’s future look like? Parents have certain ideas about the future of their children. Please rate each item by circling how important it is to you that your child attains this goal in the future.” To draw respondents’ attention to the goals they have for their children, items were rephrased to “When I think of my child, it is important to me that my child knows and accepts who he/she really is” (original item: “In the future you will know and accept who you really are”). Thus, only a new stem was added to each item. The 21 items (see Table 1) were rated on a 5-point scale ranging from 1 = not at all to 5 = very much.

To find the relative importance people placed on each subscale, mean corrected scores were computed by subtracting the total importance (the mean importance of all items) from each subscale mean (Deci & Ryan, 1997).

Child behavior problems were measured using 20 items from the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997), which were rated regarding children’s behavior difficulties over the past 6 months (from 0 = strongly disagree to 2 = strongly agree). Following Goodman, Lamping, and Ploubidis’ (2010) recommendation for analyses in low-risk samples, we differentiated the internalizing problem scale (α = .76) – indicated by the peer problems and emotional problems subscale – from the externalizing problems scale (α = .78) – indicated by the conduct problems and hyperactivity-inattention subscales, as measures of negative psychosocial adjustment.

Parental autonomy-support and psychological control were measured with the Parental Behavior Scale (Wild, 1999), an adaptation of the Children’s Report of Parental Behavior Inventory (Schaefer, 1965). Responses were made on a 4-point scale from 1 = strongly disagree to 4 = strongly agree.

The psychological control scale consists of four items and refers to demanding a high degree of conformity and compliance using enforcing rules (e.g., “I want my child to obey me immediately”; α = .76). Parental autonomy-support was measured with seven items and refers to parental acknowledgment of the child’s view and the encouragement of child-initiated activities (e.g., “I encourage my child to think about what she/he wants to see on television”; α = .66).

European Journal of Psychological Assessment (2018), 34(1), 22–29

26

J. Hollmann et al., Aspirations Index for Parental Goals for Children

Table 1. Standardized parameter estimation of the Model-1 (factor loadings) When I think of my child, it is important to me that my child...

PG

...is a very wealthy person. ...has many expensive possessions. ...has enough money to buy everything he/she wants. ...has his/her name known by many people. ...is admired by many people. ...is admired by lots of different people. ...has people commenting often (about) how attractive he/she looks. ...keeps up with fashions in hair and clothing. ...has an image that others find appealing. ...grows and learns new things. ...at the end of his/her life, is able to look back on his/her life as meaningful and complete. ...knows and accepts who he/she really is. ...shares his/her life with someone he/she loves. ...has committed, intimate relationships. ...has deep, enduring relationships. ...works for the betterment of society. ...works to make the world a better place. ...helps others improve their lives. ...feels good about his/her level of physical fitness. ...keeps herself healthy and well. ...has a physically healthy lifestyle.

Intrinsic Extrinsic

MR

CC

H

W

F

I

.724 .828 .560 .676 .853 .805 .843 .691 .695 .534 .564 .696 .538 .777 .655 .696 .816 .638

.760

.586

.548 .326

.619 .752 .656 .577 .457

.843

.954

.887

Notes. N = 691 mother, all factor loadings significant at p < .001; PG = personal growth; MR = meaningful relationship; CC = community contribution; H = health; W = wealth; F = fame; I = image.

Analyses The IBM-SPSS statistical package was used to compute descriptive statistics, internal consistency, correlations, and analysis of variance (ANOVA). Mplus 7.11 was used to compute confirmatory factor analysis (CFA) with items conceived as continuous indicators. To make use of all available data for model estimation, we used the full information maximum-likelihood estimation (FILM; Schafer & Graham, 2002). To adequately account for nonindependence of observation, the maximum-likelihood estimator with robust standard errors (MLR) and analysis option Type = Complex was used. For the model comparison, we evaluated both significance of model fit changes (tested with a log-likelihood difference test) and differences between the following fit indices: the comparative fit index (CFI, good fit > .95) and the root-mean-square error of approximation (RMSEA; good fit < .05; Hu & Bentler, 1999). Due to sensitivity of w2 to sample size, we followed the rule of thumb from Cheung and Rensvold (2002), who suggested that a decrease in CFI below .01 is an indicator of invariance. The hypothesized Model-1 was cross-validated in different subsamples, varying according to informants (mothers, fathers, mothers and fathers completing the questionnaire together) and the gender of target children, and across measurement points in time.

To test strong invariance between groups/waves, we compared unrestricted models (all parameters freely estimated) with constrained models (pattern of loadings, factor loadings, and intercepts constrained to be equal for different European Journal of Psychological Assessment (2018), 34(1), 22–29

groups/waves). Due to a small sample size in some subsamples (e.g., fathers N = 103 and both together N = 134) and missing values on cluster variables, the clustering option was changed to Type = General. We used Cronbach’s alpha as an established measurement of internal consistency. To consider the hierarchical structure of a questionnaire we also report omega. Omega can be understood as the “ratio of the true-score variance of Y to the total variance of Y”; thus, it shows to what extent the used items represent the measured factor (McDonald, 1999, pp. 89).

Results Comparison of Model-1 (w2(179) = 447.296, CFI = .930, RMSEA = .047[.041–.052]) with Model-2a (w2(180) = 485.224, CFI = .920, RMSEA = .050[.044–.055], w2(1) = 6,504.478, p < .001), with Model-2b (w2(180) = 517.611, CFI = .912, RMSEA = .052[.047–.058], w2(1) = 35.832, p < .001), and with Model-2 (w2(181) = 518.355, CFI = .912, RMSEA = .052[.047–.057], w2(2) = 174,519; p < .001) indicated that Model-1 fit the data significantly better than all other models. Model-3 failed to show acceptable goodness-of-fit indices (N = 691, w2(182) = 632.179, CFI = .882, RMSEA = .060[.055–.065]). Standardized parameter estimates for Model-1 are presented in Table 1. Factor loadings were above .35, and all items loaded significantly onto their respective factors, with R2 ranging between .29 and .73. Ó 2016 Hogrefe Publishing

J. Hollmann et al., Aspirations Index for Parental Goals for Children

27

Table 2. Descriptive statistics, Cronbach’s alpha, omega, and test-retest correlation Cronbach’s Alpha

Relative mean (SD) Scales Extrinsic: Wealth Fame Image Intrinsic: Personal growth Meaningful relationship With cross-loadings: Community-contribution Health

Overall (N = 947)

Mother (N = 691)

Father (N = 103)

Together (N = 134)

Girls (N = 433)

Boys (N = 510)

1.17 1.14 1.26 1.10 1.18 1.28 1.09

1.20 1.17 1.29 1.13 1.23 1.32 1.14

1.07 1.05 1.20 0.98 1.03 1.16 0.89

1.13 1.09 1.22 1.10 1.09 1.19 1.04

1.15 1.12 1.30 1.04 1.17 1.28 1.08

1.18 1.16 1.23 1.16 1.19 1.28 1.10

(0.45) (0.58) (0.61) (0.61) (0.49) (0.54) (0.59)

0.25 (0.62) 0.89 (0.46)

(0.41) (0.55) (0.58) (0.59) (0.45) (0.51) (0.55)

0.25 (0.62) 0.89 (0.45)

(0.52) (0.65) (0.69) (0.69) (0.59) (0.55) (0.76)

0.78 (0.64) 0.89 (0.43)

(0.50) (0.62) (0.69) (0.65) (0.57) (0.64) (0.63)

0.30 (0.55) 0.87 (0.51)

(0.43) (0.59) (0.60) (0.59) (0.49) (0.53) (0.61)

0.20 (0.60) 0.90 (0.46)

Omega

Test-retest corr.

919 < N < 936 (N = 947) (N = 947)

(0.46) (0.56) (0.62) (0.63) (0.49) (0.55) (0.58)

.896 .738 .835 .803 .721 .640 .702

.849 .761 .836 .805 .749 .659 .742

.766 .546 .506 .691 .633 .706 .474

0.30 (0.63) 0.87 (0.46)

.752 .724

.750 .742

.636 .753

Notes. For the calculation of descriptive statistics and alpha, the dimensions with cross-loading were dropped, thereafter intrinsic orientation consisted of personal growth and meaningful relationships and extrinsic orientation consisted of wealth, fame, and image.

Correlations between extrinsic and intrinsic general latent factors were low and not significant (r = .12; p = .149).

Explained variance in first-order factors ranged between .34 and .91. Taken together, these findings indicate that the questionnaire adequately reflects the hypothesized dimensions in a subsample of mothers. As expected, the assignment of the dimensions of health and community contribution to intrinsic goals should be used with caution because they carry an extrinsic meaning as well.

The model fit differed slightly between the unconstrained model (N = 928, w2(537) = 1,056.730, CFI = .917, RMSEA = .056[.051–.061]) and the constrained model (w2(621) = 1,167.685, CFI = .913, RMSEA = .053[.049– .058]) for different informants (w2(84) = 110.385, p = .028). However, the decrease in CFI was below .01.

The fit indices and exact model fit of the unconstrained model (N = 943, w2(358) = 737.150, CFI = .936, RMSEA = .047[.043–.052]) and the constrained model (w2(400) = 791.628, CFI = .934, RMSEA = .046[.041–.050]) for different groups of target children were not significantly different (w2(42) = 54.697, p = .091). Thus, multiple-group analyses confirmed invariance of models across different subsamples.

Descriptive statistics and reliabilities are shown in Table 2. The homogeneity of subscales ranged between .640 < α < .896. The scale scores contain a large amount of variance (.659 > ω < .849), reflecting a moderate to exemplary reliability (Robinson, Shaver, & Wrightsman, 1991). The hypothesized model was modeled simultaneously for T1 and T2. The unconstrained model (parameters freely estimated for T1 and T2; N = 947, w2(767) = 1,422.180, CFI = .945, RMSEA = .030[.028– .032]) did not differ significantly from a constrained model (factor and structural weight and intercepts were fixed to be equal between waves, w2(802) = 1,466.199, CFI = .944, RMSEA = .030[.027–.032], w2(35) = 44.78, p = .12). The correlation of latent scales between T1 and T2 ranged between Ó 2016 Hogrefe Publishing

.474 > r < .766, indicating an extensive to exemplary retest stability (Robinson et al., 1991).

As an indicator of scale validity, we computed an ANOVA and correlations to theoretically related variables (Table 3). A 2 3-ANOVA was performed on extrinsic parental goals and families with single parents and HISCED level in the family as between factors. As hypothesized, the more the environment threatened needs, the more important were extrinsic (compared to intrinsic) parental goals for children. There was an effect of single parent, F(1, 660) = 4.95, p < .05, and educational level, F(2, 660) = 4.81, p < .01. For educational level, planned contrasts revealed that having a lower educational level significantly increased the relative importance of extrinsic goals for children, such that there were goals for both intermediate and higher educational level, F(2, 660) = 4.8, p < .01. The single Parent Education level interaction was not significant. Spearman’s correlations indicated that the relative importance of extrinsic goals for children was associated positively with lower autonomy-supportive and more controlling strategies. Higher intrinsic parental goals related positively to autonomy-support and negatively to parental control. Higher parental extrinsic goals were associated positively with child internalizing behavior problems.

Discussion The present study tested the factorial structure and psychometric properties of an adapted and shortened version of the AI, measuring parental goals for children in a large, nonclinical sample of German parents of children between 10 and 15 years of age. Findings support the postulated structure comprising two second-order factors and seven goal-contents. Participants conceptualized the subscale and responded to the items in the same way across European Journal of Psychological Assessment (2018), 34(1), 22–29

28

J. Hollmann et al., Aspirations Index for Parental Goals for Children

Table 3. Descriptive statistics and Spearman correlation between hypothesized outcomes and parental goals for their children Relative mean (SD) Correlations

Extrinsic (1) Intrinsic (2) Autonomy-support (3) Psychological control (4) Internalizing problems (5) Externalizing problems (6)

(3)

(4)

(5)

.21 .13

.15 .19

.18 .16

.17 .19 .19

.16 .12

.34

Single parent (6) .05ns .04ns

Yes 1.10 (0.46)

HISCED No

1.22 (0.41)

Low 0.96 (0.57)

Medium 1.18 (39)

High 1.24 (0.42)

Notes. All correlations are significant at p < .01 (661 < N < 670). ns = not significant. The relative extrinsic orientation was computed by subtracting the mean importance of all intrinsic and extrinsic items from extrinsic subscale-mean (consisted of wealth, fame, and image) and the relative intrinsic orientation was computed by subtracting the mean importance of all items from intrinsic subscale-mean (consisted of personal growth and meaningful relationships).

different informants, child gender, and after an interval of about 12 months. Results concerning relationships of the scale with need-threatened environment, parental practices, and child internalizing behavior problems added to the scale’s construct validity. As goals measured using the AI have been found to be relatively stable across cultures (e.g., Grouzet et al., 2005) – and we only needed to change the stem of the items – our adaption should be transferable across different languages.

Results demonstrate that extrinsic goals and intrinsic goals are distinct constructs. Thus, parents may pursue multiple intrinsic and extrinsic parental goals for children simultaneously. Goals related to community contribution and health have been found to be both extrinsic and intrinsic. As has been reported before, health-related goals show a cross-loading on extrinsic orientation, which was almost as high as on intrinsic orientation (e.g., Klusmann et al., 2005). Parents may have understood this factor to be both intrinsic and extrinsic because the items refer to physical fitness, which is important for attractive appearance and social recognition in today’s society. The significant crossloading of the goals related to community contribution has not been reported before. However, drawing on the distinction between intra- and interpersonal perspective, community-contribution parental goals may well be perceived to be extrinsic because they may not fulfill children’s basic needs. Previous studies reported a small correlation between intrinsic and extrinsic goal orientation, which we did not find in our analysis (Klusmann et al., 2005). This correlation may be due to cross-loadings of, for example, community-contribution goals, which have been specified in our model. Future studies should explore whether the specification of such cross-loadings may improve the model for personal goals. The correlations of latent scales between time points indicate the stability of parental goals for children over the course of one year. In addition, results show homogeneity and reliability in each of the specific and overall scales, with only the values for personal growth being slightly lower, which is consistent with previous studies European Journal of Psychological Assessment (2018), 34(1), 22–29

(Klusmann et al., 2005). As expected, reliability indicators across almost all scales were higher when using omega because omega represents the highest bond, and alpha the lowest bond, of the real reliability (Brunner, Nagy, & Wilhelm, 2012).

Based on SDT, we expected that intrinsic parental goals for children motivate parental behavior that takes the child’s abilities and interests into account and has more potential for need-satisfaction. Conversely, threatened parents pursue extrinsic goals, use controlling practices, and cause the child’s introjected regulation and internalizing behavior problems. ANOVA and correlations analysis showed expected associations. The relationships were significant although the effect sizes were rather small (Cohen, 1988). Moreover, the present findings have to be interpreted in light of the sample composition. The cluster option had to be given up for model identification reasons.

In the study, the rate of participation for mothers was 73%.

We could confirm the invariance across informants, but the relative proportion of fathers (or mother and father together) was too small for generalizing the results.

Although aimed at a socially stratified sample, the sample was disproportionally staffed by families from higher track schools with presumably higher levels of education. Future studies should take different background characteristics of parents into account and explore further aspects of validity.

In addition, research should provide more information on convergent and discriminant validity of the AI-PG, gaining information from different sources and comparing different measures to assess parental goals for children as well as parental well-being, parental behaviors, and child outcomes. Despite these limitations, the AI-PG can be considered to be a reliable, valid, and economic instrument to assess parental goals for children.

Acknowledgments This research was funded by the German Federal Ministry of Education and Research (BMBF), Support Code: 01 GJ Ó 2016 Hogrefe Publishing

J. Hollmann et al., Aspirations Index for Parental Goals for Children

0983/84. We thank all schools’ executives for their support and students along with their families for their participation. We thank Anastasia Byler for native speaker advice.

References Benish-Weisman, M., Levy, S., & Knafo, A. (2013). Parents differentiate between their personal values and their socialization values: The role of adolescents’ values. Journal of Research on Adolescence, 23, 614–620. doi: 10.1111/ jora.12058 Brunner, M., Nagy, G., & Wilhelm, O. (2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80, 796–846. doi: 10.1111/j.1467-6494.2011.00749.x Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-offit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255.

Chng, G. S., Wild, E., Hollmann, J., & Otterpohl, N. (2014).

Children’s evaluative skills in informal reasoning: The role of parenting practices and communication patterns. Learning, Culture and Social Interaction, 3, 88–97.

Cohen, J. (1988). Statistical power for the behavioral sciences.

Hillsdale, NJ: Erlbaum.

Deci, E. L., & Ryan, R. M. (1997). Aspirations index. Scale Description. Retrieved from http/selfdeterminationtheory.org/ aspirations-index/ Deci, E. L., & Ryan, R. M. (2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior.

Psychological Inquiry, 11, 227–268. doi: 10.1207/ S15327965PLI1104_01 Duriez, B., Soenens, B., & Vansteenkiste, M. (2007). In search of the antecedents of adolescent authoritarianism: The relative contribution of parental goal promotion and parenting style dimensions. European Journal of Personality, 21, 507–527. doi: 10.1002/per.623 Goodman, A., Lamping, D., & Ploubidis, G. (2010). When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the Strengths and Difficulties Questionnaire (SDQ): Data from British parents, teachers and children. Journal of Abnormal Child Psychology, 38, 1179–1191. doi: 10.1007/s10802-010-9434-x Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. doi: 10.1111/j.1469-7610.1997.tb01545.x Grouzet, F. M. E., Kasser, T., Ahuvia, A., Dols, J. M. F., Kim, Y., Lau, S., . . . Sheldon, K. M. (2005). The structure of goal contents across 15 cultures. Journal of Personality and Social Psychology, 89, 800–816.

Grusec, J. E., & Goodnow, J. J. (1994). Impact of parental discipline methods on the child’s internalization of values: A reconceptualization of current points of view. Developmental Psychology, 30, 4–19. doi: 10.1037/0012-1649.30.1.4 Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi: 10.1080/10705519909540118 Kasser, T., & Ryan, R. M. (1993). A dark side of the American dream: Correlates of financial success as a central life aspiration. Journal of Personality and Social Psychology, 65, 410–422. doi: 10.1037/0022-3514.65.2.410 Kasser, T., & Ryan, R. M. (1996). Further examining the American dream: Differential correlates of intrinsic and extrinsic goals.

Ó 2016 Hogrefe Publishing

29

Personality and Social Psychology Bulletin, 22, 280–287. doi: 10.1177/0146167296223006 Kasser, T., Ryan, R. M., Zax, M., & Sameroff, A. J. (1995). The relations of maternal and social environments to late adolescents’ materialistic and prosocial values. Developmental Psychology, 31, 907–914. doi: 10.1037/0012-1649.31.6.907 Klusmann, U., Trautwein, U., & Lüdtke, O. (2005). Intrinsische und extrinsische Lebensziele [Intrinsic and extrinsic personal goals: Reliability and validity of a German translation of the Aspirations index]. Diagnostica, 51, 40–51. doi: 10.1026/00121924.51.1.40 Malmberg, L.-E., Ehrman, J., & Lithén, T. (2005). Adolescents’ and parents’ future beliefs. Journal of Adolescence, 28, 709–723.

Massey, E. K., Gebhardt, W. A., & Garnefski, N. (2008). Adolescent goal content and pursuit: A review of the literature from the past 16 years. Developmental Review, 28, 421–460.

McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.

Niemiec, C. P., Lynch, M. F., Vansteenkiste, M., Bernstein, J., Deci, E. L., & Ryan, R. M. (2006). The antecedents and consequences of autonomous self-regulation for college: A self-determination theory perspective on socialization. Journal of Adolescence, 29, 761–775.

Robinson, J. P., Shaver, P. R., & Wrightsman, L. S. (1991). Criteria for scale selection and evaluation. Measures of Personality and Social Psychological Attitudes, 1, 1–16.

Schaefer, E. S. (1965). Children’s reports of parental behavior: An inventory. Child Development, 36, 413–424.

Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147.

Sigel, I. E., & McGillicuddy-DeLisi, A. (2002). Parent beliefs are cognitions: The dynamic belief systems model. In M. B. Bornstein (Ed.), Handbook of parenting: Vol. 3: Being and becoming a parent (pp. 485–508). Mahwah, NJ: Erlbaum.

Vansteenkiste, M., Niemiec, C. P., & Soenens, B. (2010). The development of the five mini-theories of self-determination theory: An historical overview, emerging trends, and future directions. In Advances in Motivation and Achievement. The Decade Ahead: Theoretical Perspectives on Motivation and Achievement (pp. 105–165). Bingley: Emerald Group Publishing.

Vansteenkiste, M., & Ryan, R. M. (2013). On psychological growth and vulnerability: Basic psychological need satisfaction and need frustration as a unifying principle. Journal of Psychotherapy Integration, 23, 263–280.

Wild, E. (1999). Elterliche Erziehung und schulische Lernmotivation [Parenting and learning motivation.] (Unpublished habilitation thesis). Mannheim University, Germany.

Received July 14, 2014 Revision received June 8, 2015 Accepted July 11, 2015 Published online April 22, 2016

Jelena Hollmann Department of Psychology Educational Psychology Bielefeld University PO Box 100131 33501 Bielefeld Germany Tel. +49 521 106-4523 E-mail jelena.hollmann@uni-bielefeld.de

European Journal of Psychological Assessment (2018), 34(1), 22–29

Original Article

Are Workaholism and Work Engagement in the Eye of the Beholder?

A Multirater Perspective on Different Forms of Working Hard Greta Mazzetti,1 Wilmar B. Schaufeli,2,3 and Dina Guglielmi1 1

Department of Educational Science, University of Bologna, Italy

2

Department of Psychology, Utrecht University, The Netherlands

3

Faculty of Psychology and Educational Sciences, University of Leuven, Belgium Abstract: Virtually all studies on workaholism and engagement – two forms of heavy work investment – rely on self-part questionnaires.

However, the limitations of self-reports are widely acknowledged and in their final sections, papers on workaholism and engagement typically lament the use of such measures. Investigating data other than respondents’ self-reports, such as peer ratings, may overcome these limitations. Using a sample of 73 dyads composed of focal workers and their colleagues, the present study aimed: (1) to compare focal workers’ and coworkers’ perceptions concerning their levels of work engagement and workaholism; and (2) to explore the discriminant validity of engagement and workaholism. A multitrait-multimethod (MTMM) matrix and a correlated trait-correlated method model, the CT-C(M-1) model, were examined. Our results showed a considerable agreement between both raters (i.e., focal workers and coworkers) in levels of engagement and workaholism. In contrast to previous findings, a significant difference between raters on the cognitive dimension of workaholism (i.e., working compulsively) was observed. Moreover, our results provided further evidence for the discriminant validity between the two forms of heavy work investment.

Keywords: work engagement, workaholism, multitrait-multimethod analysis, CT-C(M-1) model, coworker agreement

There is a growing consensus on a conceptualization of workaholism that emphasizes the role of an overwhelming compulsion to work in order to explain the tendency to invest an excessive amount of time and energy into work (e.g., Ng, Sorensen, & Feldman, 2007). Accordingly, Schaufeli, Taris, and Bakker (2008) defined workaholism as the combination of two underlying dimensions: working excessively and working compulsively. The behavioral component of the construct – working excessively – indicates that workaholics work beyond what would be reasonably expected to fulfill organizational or economic requirements; whereas the cognitive dimension of the construct – working compulsively – implies that workaholics are obsessed with their work and persistently think about work. The motivational dynamic that propels workaholic employees to work extremely hard is referred to as controlled motivation (Van Beek, Taris, & Schaufeli, 2011). These employees are driven by the desire to avoid disapproval by others and to obtain their appreciation and, at the same time, they strive to meet high standards derived by internalization processes of external standards of self-worth and social approval European Journal of Psychological Assessment (2018), 34(1), 30–40 DOI: 10.1027/1015-5759/a000318

(Van Beek, Hu, Schaufeli, Taris, & Schreurs, 2012). Workaholism is related to detrimental consequences in several life spheres. Concerning the work domain, workaholics may display an impaired work performance (Gorgievski & Bakker, 2010) and recurrent interpersonal conflicts (Mudrack, 2006). In addition, they exhibit poor quality social relationship outside work (Bakker, Demerouti, Oerlemans, & Sonnentag, 2013) and considerable levels of work-home conflict (Schaufeli, Shimazu, & Taris, 2009). In addition, workaholism predicts health complaints (Andreassen, Hetland, Molde, & Pallesen, 2011) and higher levels of exhaustion (Kubota et al., 2011). Taken together, the motivational dynamics involved and the association with harmful outcomes constitute the main distinguishing feature of workaholism, representing a negative form of working hard, and work engagement, constituting a positive form of heavy work investment.

Work engagement is defined as a positive, fulfilling, work-related state of mind that consists of three interrelated dimensions: vigor, dedication, and absorption (Schaufeli, Salanova, González-Romá, & Bakker, 2002). Ó 2016 Hogrefe Publishing

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

Vigor is characterized by high levels of energy and mental resilience while working, the willingness to invest effort in one’s work, and persistence in the face of difficulties.

Dedication refers to being strongly involved in one’s work and experiencing a sense of significance, enthusiasm, inspiration, pride, and challenge. Finally, absorption is characterized by being fully concentrated on and happily engrossed in one’s work, whereby time passes quickly and one has difficulties with detaching oneself from work.

In contrast to workaholics, engaged employees are intrinsically motivated, so they experience their work as inherently interesting, enjoyable, and satisfying (Van Beek et al., 2011). This type of motivation encourages individuals to engage in an activity for its own sake and to act on a sense of volition; engaged employees invest a great amount of time working because they cherish this activity and have integrated their work goals mentally, which means that they are happily engrossed in their work (Van Beek et al., 2012). The positive nature of this condition is confirmed by the association of engagement with several positive outcomes. Concerning the work domain, engaged employees are more likely to craft their jobs in ways that lead to increased resources (Bakker, Albrecht, & Leiter, 2011), causing a better performance (Bakker & Bal, 2010) and lower rates of sickness absence (Schaufeli, Bakker, & Van Rhenen, 2009). Contrary to workaholics, engaged employees exhibit better social functioning outside work (Schaufeli, Taris, & Van Rhenen, 2008), enhanced engagement levels of wellbeing (Hakanen & Schaufeli, 2012), and life satisfaction (Shimazu, Schaufeli, Kubota, & Kawakami, 2012).

In conclusion, the underlying work motivation of engaged and workaholic employees differs fundamentally. The former are primarily intrinsically motivated, so they enjoy their work and feel fulfilled, whereas the latter are primarily driven by internalized standards of self-worth and social approval (Van Beek et al., 2012).

Moreover, the different nature of these conditions is confirmed by the reverse association with outcomes pertaining to the work domain, life outside work (i.e., extra job activities and social relationships), and several indicators of individual health and well-being. Finally, psychometric studies indicate that these two forms of heavy work investment can be measured independently of each other (Schaufeli, Shimazu, & Taris, 2009), although some overlap exists. Notably, confirmative factor-analytic studies indicated that the work engagement dimension classified as absorption shows a substantial double-loading on workaholism (Schaufeli, Taris, & Van Rhenen, 2008).

This overlap reflects the theoretical notion that both workaholics and engaged workers are deeply immersed in their work and are reluctant to disengage from it, albeit that their motivation differs fundamentally. Ó 2016 Hogrefe Publishing

31

A Multirater Approach to Workaholism and Engagement Over the last two decades, several scholars have drawn attention to conflicting results obtained from self-report research (e.g., Donaldson & Grant-Vallone, 2002). To be specific, social desirability, fear of negative consequences, sensitivity of constructs under investigation, and dispositional characteristics may compromise research findings exclusively based on self-report. Hence, typically in their final sections, papers on workaholism and engagement lament the use of self-report measures for these very reasons. Collecting data other than respondents’ self-reports – e.g., peer assessments – may potentially overcome these problems.

Porter (1996) speculated that workaholics are often unaware of the obsession that leads them to be completely immersed in their work. Because of this tendency to deny, workaholics’ evaluation of their behavior and their attitude toward work might not agree with their significant others’ views; thus, they may underestimate their obsession with work. Moreover, they may be unconscious of the damaging effects that long working hours may have on their physical and psychological well-being. Accordingly, it may be argued that coworkers, who spend the majority of their working day in close contact with workaholic employees, acknowledge their heavy work investment and its detrimental outcomes. Porter (1996) points to the parallel that exists with other types of addicts, for instance, drug addicts and alcoholics also tend to deny that they are addicted, and refuse treatment. This evokes the original conceptualization that described workaholism as a veritable kind of addiction and that emphasized its similarity with alcoholism (Oates, 1971).

To date, few studies have addressed the claim that workaholics deny and therefore underreport their compulsive conduct by gathering data from more than one source.

The first study, conducted by McMillan, O’Driscoll, and Brady (2004), collected data from both focal workers (N = 88) and their partners (N = 40). Participants completed two scales of the Workaholism Battery (WorkBat; Spence & Robbins, 1992) – feeling driven to work and work enjoyment – and estimated the number of hours they worked per week. The results indicated that workaholic employees (i.e., the focal person) rated their work enjoyment slightly higher than their partners did. Most surprisingly, workaholics rated themselves significantly higher in drive than their partners. Accordingly, workaholics did not tend to underreport their compulsive conduct toward work in comparison to their partners.

In a similar vein, Aziz and Zickar (2006) assessed the level of agreement on the three workaholism dimensions distinguished by Spence and Robbins (1992) – work European Journal of Psychological Assessment (2018), 34(1), 30–40

32

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

involvement, feeling driven to work, and work enjoyment – between focal workers and an acquaintance (i.e., family member, friend, or coworker). Analyses were based on a total of 174 paired surveys and revealed that the acquaintances substantiated the responses provided by the focal workers. In other words, the study found comparable mean ratings between acquaintances’ and focal workers’ responses.

Burke and Ng (2007) collected data from workers in professional and managerial jobs (N = 62) along with a selfnominated coworker. Akin to the previous study, the obtained results showed a substantial agreement on all three components of workaholism. Moreover, this study’s participants (i.e., focal workers and their colleagues) showed comparable scores on a one-item global assessment of workaholism.

On the whole, the previous findings provide evidence for a substantial agreement among self-reports and the ratings provided by significant others, signifying that focal workers do not tend to deny their behavior. However, all previous studies were based on the workaholic triad developed by Spence and Robbins (1992), which distinguishes between positive and negative forms of workaholism resulting from different combinations of three dimensions: work involvement, drive, and work enjoyment. In contrast, in the current study workaholism is defined as a negative psychological state characterized by working excessively due to an irresistible inner drive (Schaufeli, Taris, & Bakker, 2008).

Whereas research on workaholism has tried to gather data from multiple sources in order to evaluate the differences between self-reports and significant others’ reports, to the best of our knowledge, the present research represents the first attempt to evaluate multirater agreement on work engagement. This is interesting since this positive state may transfer from one individual to another both in the work environment as well as in the family context. This process that occurs when the psychological well-being experienced by one person affects the level of well-being of another person, is referred to as crossover (Westman, 2001). Previous research has provided evidence for a reciprocal crossover of engagement (i.e., vigor and dedication) among partners (Bakker & Demerouti, 2009). Work engagement is also contagious within work teams, so that team-level engagement is related to individual members’ engagement (Bakker, Van Emmerik, & Euwema, 2006).

More specifically, engagement transmits from one employee to another, particularly on days when coworkers interact more frequently than usual (Bakker & Xanthopoulou, 2009). Although the level of engagement exhibited by employees has a relevant and beneficial impact on the motivation and the attitude toward work experienced by coworkers, research on others’ perceptions of this work-related condition is still lacking.

European Journal of Psychological Assessment (2018), 34(1), 30–40

Therefore, the purpose of the present study is twofold.

On the one hand, it aims to compare focal workers’ and their colleagues’ perceptions concerning focal workers’ level of workaholism, as measured by the Dutch Work Addiction Scale (DUWAS; Schaufeli, Shimazu, & Taris, 2009), and work engagement, as measured with the Utrecht Work Engagement Scale (UWES; Schaufeli, Bakker, & Salanova, 2006). On the other hand, both measures will also be employed to investigate the discriminant validity of work engagement and workaholism using different information sources.

Materials and Methods Participants Focal Workers The participants consisted of 73 dyads of Italian workers.

The focal workers were mostly female (53.4%), the mean age was 41.16 years (SD = 6.51), 61.6% of participants worked in the commercial sector, 28.8% in the industrial sector, and the remaining 9.6% worked in public administration. The majority of the sample worked as employees (50.8%), had a permanent job (95.9%) with a full-time contract (97.3%), and the mean organizational seniority was 10.77 years (SD = 7.1).

Coworkers The slight majority of coworkers were women (58.9%), the mean age was 36.14 years (SD = 7.60). Sixty-one percent of coworkers worked in the commercial sector, 28.8% in the industrial sector, and 9.6% in public administration. The majority of this group worked as employees (60.6%), had a permanent job (87.7%), worked full time (94.5%), and had an organizational seniority of 6.73 years (SD = 5.98).

Procedure Questionnaires were distributed to 73 individuals working for five different Italian organizations operating in several occupational sectors as part of an occupational health survey. These focal workers were randomly selected by the research group and were provided with two copies of the same questionnaire. Each focal worker had to complete one of these copies as a self-report questionnaire, and was instructed to have the other copy filled out by a coworker who habitually worked with him/her. Both the copies of the questionnaire included a cover letter that provided participants with background information about the general aim of the study. In the introduction to the survey, participant anonymity was emphasized and confidentiality guaranteed. Ó 2016 Hogrefe Publishing

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

In order to obtain the other assessment of the level of work engagement and workaholism displayed by the focal worker, the coworker version of the questionnaire was adapted by reformulating all the items into a “he/she” version, and by modifying the instructions accordingly.

Therefore, the answers provided by both subjects constitute a dyad, which pertains to the same subject, namely the focal worker. After completion, the coworkers put their questionnaire in a sealed envelope and returned it to the focal worker.

Measures Work engagement was assessed using the nine-item version of the Utrecht Work Engagement Scale (Schaufeli et al., 2006; Italian version: Balducci, Fraccaroli, & Schaufeli, 2010), which includes three subscales of three items each: vigor, dedication, and absorption. Focal workers answered items such as: “At my job, I feel strong and vigorous” (vigor); “My job inspires me” (dedication); and “I am immersed in my job” (absorption). The coworker version of the questionnaire was reworded in order to reflect their experience of the focal workers’ level of work engagement (e.g., “In his/her job, my colleague feels strong and vigorous”).

All items were scored on a 7-point rating scale ranging from 0 = never to 6 = always.

Workaholism was assessed using the 10-item Dutch Work Addiction Scale (DUWAS; Schaufeli, Shimazu, & Taris, 2009; Italian version: Molino, 2012), which included two subscales of five items: working excessively and working compulsively. Example items for the focal worker version of the questionnaire are: “I stay busy and keep my irons in the fire” (working excessively); “I feel guilty when I take time off work” (working compulsively). The coworkers’ version of the scale was adapted in order to understand their perspective on focal workers’ level of workaholism, therefore all the items were reformulated accordingly (e.g., “My colleague stays busy and keeps his/her irons in the fire” and “My colleague feels guilty when he/she takes time off work”).

All items were rated on a 4-point frequency scale ranging from 1 = never to 4 = always.

Strategy of Analysis The two main purposes of the study were achieved by using two different strategies of analysis: first, the MultitraitMultimethod (MTMM) matrix (i.e., correlations among measures of multiple traits assessed by multiple methods) 1

33

provided preliminary information about the convergent and discriminant validity between work engagement and workaholism (Campbell & Fiske, 1959).

Next, a multiple-indicator correlated trait-correlated method minus one model, namely the so-called CT-C(M-1) model, was analyzed (Eid, Lischetzke, Nussbeck, & Trierweiler, 2003; Nussbeck, Eid, Geiser, Courvoisier, & Lischetzke, 2009). The CT-C(M-1) model is a special case of the correlated trait-correlated method model, with one method factor less than the numbers of methods considered. Indeed, in this model one of the methods is selected as a reference method (or standard method), and is not modeled as a factor.

In the current model, we had structurally different methods, since each of them had a particular perspective on the focal person. In other words, the focal worker was asked to rate him- or herself, while the coworker was asked to rate the focal worker. The presence of structurally different methods led us to define focal worker (i.e., self-report) as the reference method. The exclusion of a specified method factor for self-report implied that the trait factors (engagement and workaholism) were interpreted as the traits measured by focal workers. Hence, the present study contrasted the focal worker self-report with the coworker report, thus it explored the deviations of the coworker ratings from the self-report ratings provided by the focal worker.

Results Descriptive Statistics Table 1 displays the full MTMM matrix of the correlations among the components of engagement and workaholism and the total scores on these dimensions, as measured by focal workers and coworkers.

Overall, a comparison of the means of the focal workers and coworkers revealed very similar patterns. Nonetheless, the focal workers’ average self-evaluations were in general slightly higher than those provided by their coworkers for the two central dimensions of engagement, namely vigor and dedication. However, this difference was not statistically significant either for vigor, t(72) = .77, p = .45, d1 = .08, or for dedication, t(72) = .97, p = .34, d = .10. In contrast, focal workers’ assessment of absorption was not only higher than the average rating provided by coworkers, but this difference was also far from being significant, t(72) = .18, p = .86, d = .03. Finally, the focal workers’ average self-evaluations on the general score of work engagement were higher than those provided by their

To avoid overestimation, Cohen’s d for the paired t-tests was calculated using original standard deviations instead of pooled standard deviations (Dunlop, Cortina, Vaslow, & Burke, 1996).

Ó 2016 Hogrefe Publishing

European Journal of Psychological Assessment (2018), 34(1), 30–40

34

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

Table 1. Means, standard deviation, Cronbach’s alphas (in parentheses), and MTMM correlations among the variables r Self report Method and trait Self report 1. Vigor 2. Dedication 3. Absorption 4. Work engagement 5. WE 6. WC 7. Workaholism Coworker report 1. Vigor 2. Dedication 3. Absorption 4. Work engagement 5. WE 6. WC 7. Workaholism

M

SD

1

2

3

5.19 5.49 5.40 5.36

0.76 0.84 0.57 0.62

(.81) .73*** .54*** .90***

(.92) .47*** .89***

(.60) .74***

2.76 0.64 .26* 2.25 0.57 .27* 2.50 0.58 .28* 5.11 5.40 5.42 5.31

1.04 0.92 0.74 0.83

.58*** .52*** .42*** .56***

2.78 0.49 .18 2.42 0.55 .07 2.60 0.47 .13

.30** .26* .30* .52*** .62*** .51*** .60*** .11 .09 .11

.15 .14 .15 .33** .24* .17 .28* .09 .16 .14

Coworker report

4

5

6

(.83) .82*** .95***

(.73) .95***

7

1

2

3

4

5

6

7

(.88) .29* .27* .29* .57*** .57*** .46*** .59*** .15 .12 .15

.21 .20 .19 .22 .70*** .46*** .64***

.12 .12 .19 .15 .67*** .46*** .62***

(.88) .18 .17 .20 .20

(.83) .79*** (.91) .67*** .84*** (.76) .91*** .95*** .89***

.72*** .08 .48*** .03 .66*** .02

.06 .21 .15

.02 .20 .12

(.92) .01 (.64) .15 .64*** (.66) .08 .89*** .91*** (.78)

Notes. *p < .05; **p < .01; ***p < .001. Coefficient alpha is displayed in parentheses on the main diagonal. Correlations between the same trait measured by two different methods (convergent validity) are in bold.

coworkers. Again, the comparison between these means revealed that they were not significantly different, t(72) = .61, p = .55, d = .07.

The assessments of the workaholism dimensions indicated that the scores provided by coworkers were higher than those provided by focal workers. In particular, there was a significant difference for working compulsively, t(72) = 2.50, p = .01, d = .31, with coworkers assigning higher scores than focal workers. In contrast, focal workers’ assessment of working excessively was not significantly different from the average rating provided by coworkers, t(72) = .46, p = .65, d = .03. In a similar vein, the focal workers’ average self-evaluations on the general score of workaholism were lower than those provided by coworkers, but these average ratings were not significantly different, t(72) = 1.87, p = .07, d = .19. Therefore, the only significant difference was observed in the cognitive component of workaholism, whereby focal workers report lower levels of their own compulsive work behavior compared to their coworkers.

Correlation Coefficients Among MultitraitMultimethod (MTMM) Measures In the MTMM matrix, high correlations between measures of the same trait assessed by different methods provide evidence of convergent validity. Conversely, discriminant validity is supported if correlations among measures of different traits (using either the same or different methods) are significantly weaker than correlations between measures of the same trait provided by different methods.

European Journal of Psychological Assessment (2018), 34(1), 30–40

An inspection of the MTMM correlations from Table 1 revealed a significant convergence between focal workers and coworkers in all the reported components (rs ranged from .46 to .70). The only exception is constituted by absorption, which showed a nonsignificant correlation between focal worker- and coworker reports. Our results also supported the convergent validity of the general scores of work engagement (r = .59, p < .001) and workaholism (r = .66, p < .001).

Overall, these correlations support the convergent validity for the dimensions of engagement and workaholism as well as their composite scores, with the exception of the absorption component of engagement. When we take the single dimensions of engagement and workaholism into account, we found evidence for the prevalence of strong method effect: in several cases, correlations among different dimensions measured by the same method were higher than the respective convergent validity coefficients. However, when the total engagement score is considered, the highest correlation by far is that between the heteromethod-monotrait measures, hence between engagement as measured by focal worker and coworker (r = .59, p < .001). Therefore, work engagement showed strong discriminant validity from the general score of workaholism and its components, specifically working excessively and working compulsively. In a similar vein, when only the general workaholism score is assessed, the highest correlation is between the heteromethod-monotrait measure, hence between workaholism as assessed by the focal worker and its coworker (r = .66, p < .001). Hence, these results provided evidence of the discriminant validity from work engagement and its dimensions, namely vigor, dedication, and absorption. Ó 2016 Hogrefe Publishing

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

35

Figure 1. CT-C(M-1) model for work engagement and workaholism with self-report as reference method.

WE = Working Excessively; WC = Working Compulsively.

VIGOR

DEDICATION

ABSORPTION WORK ENGAGEMENT

VIGOR

COWORKER

DEDICATION

REPORT ENGAGEMENT

ABSORPTION

WE

WC WORKAHOLISM

WE COWORKER REPORT WORKAHOLISM

WC

Testing the CT-C(M-1) Model The CT-C(M-1) model was estimated using the AMOS 5 software package (Arbuckle, 2005) with a maximum likelihood estimation procedure (Figure 1).

The model fit to the data was evaluated using the chisquare (w2) statistic and the root mean square error of approximation (RMSEA). We also examined fit indices less sensitive to sample size, including the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI). For the RMSEA, values less than or equal to .08 indicate an acceptable model fit (Bentler, 1990). For the other fit statistics, values of .90 represent acceptable fit, whereas values of .95 or higher indicate good fit (Hu & Bentler, 1999).

The model presented in Figure 1 showed a good fit to the data: w2 (28) = 36.137, p = .14; RMSEA = .06, CFI = .98, and TLI = .97.

Standardized factor loadings for the trait and method factors are reported in Table 2.

Factor loadings were strong (.58–.94) for focal worker reports as well as for coworker reports (.50–.75). Coworker report represented the only modeled method factor of the Ó 2016 Hogrefe Publishing

current study, since focal worker report was selected as reference method it was therefore not modeled. The high loadings of coworker reports on the trait factors indicate that focal worker reports explain a large amount of the variances of their coworkers’ ratings. Consequently, these coefficients provided additional support to the convergent validity between focal worker reports (i.e., self-reports) and coworker reports.

There was some variability across traits: the lowest trait loadings for coworker reports were found for absorption and working compulsively (.50), whereas working excessively showed the highest loading (.75). This suggests that convergent validity of focal worker report vis-à-vis coworker report was strongest for working excessively, and weakest for absorption and working compulsively.

In the CT-C(M-1) model, the correlation of different traits measured by the same method indicates the generalizability of method effects across traits (Eid et al., 2003). Hence, a correlation of 0 would indicate that there is no generalizability of method effects across traits, whereas a correlation equal to 1 would suggest a perfect homogeneity of method effects across traits. In the current study, the correlation

European Journal of Psychological Assessment (2018), 34(1), 30–40

36

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

Table 2. Standardized factor loadings for trait and method (self-report and coworker report) factors from the CT-C(M-1) model

Self-report

Coworker report

Work engagement

Workaholism

Method

Method

Indicator

Trait

Vigor Dedication Absorption Vigor Dedication Absorption

.87*** .85*** .58*** .63*** .62*** .50***

Coworker report

.50*** .78*** .66***

Indicator

Trait

WE WC

.94*** .87***

WE WC

.75*** .50***

Coworker report

.40*** .67**

.33**

Correlation between traits (work engagement, workaholism) Correlation between methods (coworker report of work engagement, coworker report of workaholism)

.54**

Note. WE = Working Excessively; WC = Working Compulsively. **p < .01; ***p < .001.

between engagement and workaholism as measured by coworkers was r = .54 ( p = .02). This positive correlation indicates that coworkers who overestimate focal worker engagement also tend to overestimate that person’s level of workaholism. Conversely, underestimation of engagement is associated with underestimation of workaholism.

The correlation of the trait factors (i.e., work engagement and workaholism) indicates the discriminant validity at the level of the standard method. As reported in Table 2, work engagement showed a negative correlation with workaholism (r = .33, p = .01). This negative correlation coefficient indicates that engagement and workaholism emerge as distinct dimensions. Table 3 illustrates the variance components of the observed variables and the true-score variables.

All the variance components have been computed using the formulas indicated by Eid and colleagues (2003).

The reliabilities of the observed indicators are relatively high, with the exception of absorption as assessed by focal workers. In line with the internal consistency indicated in Table 1, the reliability of this specific indicator is slightly lower than the value of .70 which is generally used as an indicator for sufficient internal consistency (Nunnally & Bernstein, 1994). The consistency coefficient describes the amount of true variance for an observed variable, or true-score variables, which is explained by trait factors.

In other words, it indicates the degree to which true differences between the ratings are due to differences between targets (and not due to differences between raters). The method-specificity coefficient, on the other hand, reveals the amount of variance of an observed variable, or truescore variable, as assessed by a non-reference method (i.e., coworker reports) that is specific to this method factor.

As a consequence, this coefficient cannot be computed for the reference method (i.e., focal worker). In other words, it represents the proportion of true variance of the assessments that is due to differences between raters (and not due to differences between targets).

European Journal of Psychological Assessment (2018), 34(1), 30–40

For the three work engagement indicators (i.e., observed variables), the consistency coefficients of the coworkers’ ratings range from .26 to .61. Hence, between 26% and 61% of the coworkers’ ratings can be explained by the focal worker reports. Inspection of the method-specificity coefficients of the engagement dimensions suggests that between 25% and 44% of reliable variation in these dimensions as reported by coworkers was unique to this method.

When the variance components of the true-score variables (i.e., latent variables) are considered, the consistency coefficients of the coworker ratings for vigor and dedication range from .61 to .62; therefore, a preponderance of method specificity can be excluded. In line with the data presented in Table 1, the consistency coefficient of the coworker rating for the latent absorption is equal to .37, whereas the method specificity coefficient is equal to .63, suggesting that 63% of reliable variation in absorption as evaluated by coworkers was unique to this method.

The consistency coefficients of the two indicators of workaholism of the coworkers’ ratings range from .24 to .56. When the variance components of the true-score variables are examined, the consistency coefficient of the coworker ratings for working excessively is .78, thus suggesting a strong association between focal worker reports and coworker reports for this dimension of workaholism.

Working compulsively showed a consistency coefficient of .35, and a method specificity coefficient of .65, suggesting that 65% of reliable variation in working compulsively as assessed by coworkers’ was unique to this method.

The last column of Table 3 shows the correlations between the true scores of the coworker ratings and the corresponding true scores of the first focal worker (i.e., self-reported) indicator. Hence, they represent correlations between focal worker- and coworker reports corrected for measurement error. Thus, the latent correlations between coworker-reported latent traits and focal worker indicators ranged from .59 to .88. Ó 2016 Hogrefe Publishing

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

37

Table 3. Variance components in the CT-C(M-1) model Observed variables Rating Work engagement Self-report Vigor Dedication Absorption Coworker report Vigor Dedication Absorption Workaholism Self-report Working excessively Working compulsively Coworker report Working excessively Working compulsively

Reliability

Consistency

0.75 0.72 0.33

.75 .72 .33

0.64 1.00 0.70

.39 .39 .26

0.88 0.75

.88 .75

0.71 0.69

.55 .24

True-score variables

Method specificity

Consistency

Method specificity

Latent correlation

.38 .61 .63

.79 .62 .61

.22 .65

.88 .59

1.00 1.00 1.00 .25 .61 .44

0.62 0.39 0.37

1.00 1.00 .15 .45

0.78 0.35

p Note. CT-C(M-1) = correlated trait-correlated method minus one. Latent correlation with the standard method ( consistency).

Discussion The present study compared focal workers’ and coworkers’ perceptions of engagement and workaholism exhibited by the focal worker, and explored the discriminant validity of these different types of working hard.

In line with the MTMM matrix, the CT-C(M-1) model supported the discriminant validity between these constructs at the level of the focal worker, chosen as reference method. In line with previous studies (Schaufeli, Taris, & Van Rhenen, 2008), the current research provided evidence for the distinctive nature of these forms of working hard.

The presence of a negative, yet not large, correlation between these constructs is consistent with psychometric studies which corroborated the hypothesis that these two forms of heavy work investment can be measured independently of each other (Schaufeli, Taris, & Bakker, 2008; Schaufeli, Shimazu, & Taris, 2009), although some overlap exists.

According to the MTMM matrix, the assessment of the three dimensions of work engagement showed a substantial agreement between the two groups of raters with the only exception of absorption. However, vigor and dedication are regarded as the core features of work engagement (Schaufeli & Bakker, 2004). Therefore, these results provided support for the convergent validity between focal worker and coworker with respect to both central dimensions of work engagement.

Work engagement as assessed by the UWES is conceived of as a unitary construct constituted by three different yet closely related aspects. As Schaufeli and colleagues (2006) recommend, the total score on the UWES should be used as a single indicator of engagement. In our study, the general assessment of engagement showed a high Ó 2016 Hogrefe Publishing

agreement between the two raters involved; therefore, these results substantiated the convergent validity between ratings of a positive kind of heavy work investment, namely work engagement.

For workaholism, the MTMM matrix indicated highly comparable evaluations for the behavioral component of the construct (i.e., working excessively), but significantly different assessments of the cognitive dimension (i.e., working compulsively). In our sample, focal workers tended to report lower levels of their compulsiveness in comparison to their coworkers. This might be explained by the different nature of the two dimensions of workaholism as described

by Schaufeli, Taris, and Bakker (2008). Working excessively refers to a type of behavior which is by definition manifest to observers: accordingly, our results indicate that the evaluation of the behavioral dimension of workaholism is more consistent among the raters involved. In contrast, working compulsively refers to the obsessive nature of the underlying motivation to work hard (Schaufeli, Shimazu, & Taris, 2009). This uncontrollable pressure toward work implies an inner drive, which cannot be directly observed by others. The cognitive component of workaholism is covert unless it translates into overt excessive work behaviors; therefore, self- and other-ratings regarding this dimension are likely to diverge significantly.

On the other hand, the total score on workaholism did not show any significant difference between the raters. This finding is particularly relevant because workaholism is defined as a syndrome, which implies the combination of high scores on both its underlying dimensions: working excessively and working compulsively (Schaufeli et al., 2009).

All in all, results concerning the convergent validity between focal worker and coworker assessments of European Journal of Psychological Assessment (2018), 34(1), 30–40

38

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

engagement and workaholism and their dimensions were corroborated by loadings and consistency coefficients reported in the CT-C(M-1) model. The high trait loadings indicated that focal worker report (i.e., self-report) explains a large amount of the variances of coworkers’ ratings. In particular, the highest loading for coworker report pertained to working excessively, whereas the lowest loading was obtained for absorption and working compulsively. In line with the MTMM matrix, the analysis of the variance components of the CT-C(M-1) model indicated that absorption and working compulsively showed strong method specificity: a large amount of variance for these dimensions was explained by method factors. Therefore, the two dimensions of workaholism exhibited a different convergent validity between focal workers’ and coworkers’ assessments. This deviates from previous findings suggesting a complete overlap between self- and other reports of the dimensions of the Spence and Robbins (1992) workaholism triad (Aziz & Zickar, 2006; Burke & Ng, 2007; McMillan et al., 2004).

On the whole, the present research corroborates the evidence that work engagement and workaholism represent two conceptually and empirically distinct forms of heavy work investment that seem to be similarly assessed by both focal workers and their coworkers.

Although a strength of the present study design is the exploration and matching of data between focal workers and their colleagues, there are some limitations that should be mentioned. First, the sample size was relatively small, which might have reduced the statistical power of our analyses and decreased the opportunity to generalize the obtained results to the entire working population.

A second limitation is the cross-sectional nature of the study, so we cannot draw any conclusions regarding the stability of our findings. A longitudinal research design would allow us to investigate whether the current results are stable across time.

Finally, the engagement dimension of absorption (as measured by focal workers) and the two dimensions of workaholism (as measured by coworkers) had a reliability coefficient slightly lower than the criterion of .70 (Nunnally & Bernstein, 1994). Nevertheless, this alpha coefficient is satisfactory considering Nunnally’s (1967) recommendation to only use scales with item consistencies higher than .60 in basic research. In addition, the current study focused on two different kinds of heavy work investment, therefore a crucial role was attributed to the total scores of the UWES and the DUWAS.

Future Research Directions Despite these limitations, the current findings have implications for future research. Indeed, for future studies on a European Journal of Psychological Assessment (2018), 34(1), 30–40

multirater perspective of workaholism and engagement, it may be of interest to investigate also the perceptions of other subjects, both within the workplace (e.g., supervisors) and the family context (e.g., partners). This will allow us to uncover the overlap or differences in focal workers’ workaholism and engagement as measured by other, different raters.

In addition, using the normative scores of the DUWAS (Schaufeli, Shimazu, & Taris, 2009), future studies should select only participants identified as being workaholic in order to specifically address the claim that these workers are unaware of their compulsive conduct toward work, and therefore tend to display a denial tendency that leads them to underestimate this obsession with work (Porter, 1996).

Moreover, future research should investigate the effective impact of focal workers’ workaholism and engagement on personal relationships. To this end, measures of relationship quality should be assessed by different raters (e.g., coworkers, partners) in order to corroborate the hypothesis that workaholism and engagement have detrimental and positive consequences also on the quality of workers’ relationships.

Final Note Recurrent problems in research on workaholism and work engagement are the self-assessment of both constructs and their discriminant validity. Based on the current study, researchers can be more confident that worker’s selfassessments agree with ratings of their colleagues, and that work engagement and workaholism tap different kinds of heavy work investment.

References Arbuckle, J. L. (2005). Amos 6.0 user’s guide. Chicago, IL: SPSS. Andreassen, C. S., Hetland, J., Molde, H., & Pallesen, S. (2011).

“Workaholism” and potential outcomes in well-being and health in a cross-occupational sample. Stress and Health, 27, e209–e214. doi: 10.1002/smi.1366 Aziz, S., & Zickar, M. J. (2006). A cluster analysis investigation of workaholism as a syndrome. Journal of Occupational Health Psychology, 11, 52–62. doi: 10.1037/1076-8998.11.1.52 Bakker, A. B., Albrecht, S., & Leiter, M. P. (2011). Key questions regarding work engagement. European Journal of Work and Organizational Psychology, 20, 4–28. doi: 10.1080/ 1359432X.2010.485352 Bakker, A. B., & Bal, P. (2010). Weekly work engagement and performance: A study among starting teachers. Journal of Occupational and Organizational Psychology, 83, 189–206. doi: 10.1348/096317909X402596 Bakker, A. B., & Demerouti, E. (2009). The crossover of work engagement between working couples: A closer look at the role Ó 2016 Hogrefe Publishing

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

of empathy. Journal of Managerial Psychology, 24, 220–236. doi: 10.1108/02683940910939313 Bakker, A. B., Demerouti, E., Oerlemans, W., & Sonnentag, S. (2013). Workaholism and daily recovery: A day reconstruction study of leisure activities. Journal of Organizational Behavior, 34, 87–107. doi: 10.1002/job.1796 Bakker, A. B., Van Emmerik, H., & Euwema, M. C. (2006).

Crossover of burnout and engagement in work teams. Work and Occupations, 33, 464–489. doi: 10.1177/ 0730888406291310 Bakker, A. B., & Xanthopoulou, D. (2009). The crossover of daily work engagement: Test of an actor-partner interdependence model. Journal of Applied Psychology, 94, 1562–1571. doi: 10.1037/a0017525 Balducci, C., Fraccaroli, F., & Schaufeli, W. B. (2010). Psychometric properties of the Italian version of the Utrecht work engagement scale (UWES-9): A cross-cultural analysis. European Journal of Psychological Assessment, 26, 143–149. doi: 10.1027/1015-5759/a000020 Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. doi: 10.1037/ 0033-2909.107.2.238 Burke, R. J., & Ng, E. W. (2007). Workaholic behaviors: Do colleagues agree? International Journal of Stress Management, 14, 312–320. doi: 10.1037/1072-5245.14.3.312 Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. doi: 10.1037/h0046016 Donaldson, S. I., & Grant-Vallone, E. J. (2002). Understanding selfreport bias in organizational behavior research. Journal of Business and Psychology, 17, 245–260. doi: 10.1023/ A:1019637632584 Dunlop, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996).

Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods,170–177. doi: 10.1037/1082-989X.1.2.170 Eid, M., Lischetzke, T., Nussbeck, F. W., & Trierweiler, L. I. (2003).

Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38–60. doi: 10.1037/1082989X.8.1.38 Gorgievski, M. J., & Bakker, A. B. (2010). Passion for work: Work engagement versus workaholism. In S.L. Albrecht (Ed.), Handbook of employee engagement: Perspectives, issues, research and practice (pp. 264–271). Glos, UK: Edward Elgar.

Hakanen, J. J., & Schaufeli, W. B. (2012). Do burnout and work engagement predict depressive symptoms and life satisfaction? A three-wave seven-year prospective study. Journal of Affective Disorders, 141, 415–424.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi: 10.1080/10705519909540118 Kubota, K., Shimazu, A., Kawakami, N., Takahashi, M., Nakata, A., & Schaufeli, W. B. (2011). Association between workaholism and sleeping problems among hospital nurses. Industrial Health, 48, 864–871.

McMillan, L. W., O’Driscoll, M. P., & Brady, E. C. (2004). The impact of workaholism on personal relationships. British Journal of Guidance & Counselling, 32, 171–186. doi: 10.1080/ 03069880410001697729 Molino, M. (2012). Workaholism: Definitions, measures, and dynamics Unpublished doctoral dissertation, University of Torino, Torino, Italia.

Ó 2016 Hogrefe Publishing

39

Mudrack, P. E. (2006). Understanding workaholism: The case of behavioral tendencies. In R.J. Burke (Ed.), Research companion to working time and work addiction (pp. 108–128). Cheltenham, UK: Edward Elgar.

Ng, T. W. H., Sorensen, K., & Feldman, D. (2007). Dimensions, antecedents, and consequences of workaholism: A conceptual integration and extension. Journal of Organizational Behavior, 28, 111–136. doi: 10.1002/job.424 Nunnally, J. C. (1967). Psychometric theory (1st ed.). New York, NY: McGraw-Hill.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

Nussbeck, F. W., Eid, M., Geiser, C., Courvoisier, D. S., & Lischetzke, T. (2009). A CTC(M-1) model for different types of raters. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5, 88–98. doi: 10.1027/ 1614-2241.5.3.88 Oates, W. (1971). Confessions of a workaholic. New York, NY: World.

Porter, G. (1996). Organizational impact of workaholism: Suggestions for researching the negative outcomes of excessive work.

Journal of Occupational Health Psychology, 1, 70–84. doi: 10.1037/1076-8998.1.1.70 Schaufeli, W. B., & Bakker, A. B. (2004). Job demands, job resources, and their relationship with burnout and engagement: A multi-sample study. Journal of Organizational Behavior, 25, 293–315. doi: 10.1002/job.248 Schaufeli, W. B., Bakker, A. B., & Salanova, M. (2006). The measurement of work engagement with a Short Questionnaire: A cross-national study. Educational and Psychological Measurement, 66, 701–716. doi: 10.1177/0013164405282471 Schaufeli, W. B., Bakker, A. B., & Van Rhenen, W. (2009). How changes in job demands and resources predict burnout, work engagement and sickness absenteeism. Journal of Organizational Behavior, 30, 893–917. doi: 10.1002/job.595 Schaufeli, W. B., Salanova, M., González-Romá, V., & Bakker, A. B. (2002). The measurement of engagement and burnout: A two sample confirmatory factor analytic approach. Journal of Happiness Studies, 3, 71–92. doi: 10.1023/A:1015630930326 Schaufeli, W. B., Shimazu, A., & Taris, T. W. (2009). Being driven to work excessively hard: The evaluation of a two-factor measure of workaholism in the Netherlands and Japan. Cross-Cultural Research, 43, 320–348. doi: 10.1177/1069397109337239 Schaufeli, W. B., Taris, T. W., & Bakker, A. B. (2008). It takes two to tango: Workaholism is working excessively and working compulsively. In R. J. Burke & C. L. Cooper (Eds.), The long work hours culture: Causes, consequences and choices (pp. 203– 226). Bingley, UK: Emerald.

Schaufeli, W. B., Taris, T. W., & Van Rhenen, W. (2008). Workaholism, burnout, and work engagement: Three of a kind or three different kinds of employee well-being? Applied Psychology: An International Review, 57, 173–203. doi: 10.1111/j.14640597.2007.00285.x Shimazu, A., Schaufeli, W. B., Kubota, K., & Kawakami, N. (2012).

Do workaholism and work engagement predict employee wellbeing and performance in opposite directions? Industrial Health, 50, 316–321. doi: 10.2486/indhealth.MS1355 Spence, J. T., & Robbins, A. S. (1992). Workaholism: Definition, measurement, and preliminary results. Journal of Personality Assessment, 58, 160–178. doi: 10.1207/s15327752jpa5801_15 Van Beek, I., Hu, Q., Schaufeli, W. B., Taris, T. W., & Schreurs, B. H. J. (2012). For fun, love, or money: What drives workaholic, engaged, and burned-out employees at work? Applied Psychology: An International Review, 61, 30–55. doi: 10.1111/j.14640597.2011.00454.x

European Journal of Psychological Assessment (2018), 34(1), 30–40

40

G. Mazzetti et al., Workaholism and Engagement in the Eye of the Beholder

Van Beek, I., Taris, T. W., & Schaufeli, W. B. (2011). Workaholic and work engaged employees: Dead ringers or worlds apart?

Journal of Occupational Health Psychology, 16, 468–482. doi: 10.1037/a0024392 Westman, M. (2001). Stress and strain crossover. Human Relations, 54, 717–752. doi: 10.1177/00187267015460 Received May 20, 2014 Revision received June 9, 2015 Accepted July 11, 2015 Published online April 22, 2016

European Journal of Psychological Assessment (2018), 34(1), 30–40

Greta Mazzetti Department of Educational Science University of Bologna Via Filippo Re, 6 40126 Bologna Italy Tel. +39 051 2091622 Fax +39 051 2091474 E-mail greta.mazzetti3@unibo.it

Ó 2016 Hogrefe Publishing

Psychological Assessment Science and Practice Book series Edited with the support of the European Association of Psychological Assessment (EAPA) Editors Tuulia M. Ortner, PhD, Austria Itziar Alonso-Arbiol, PhD, Spain Anastasia Efklides, PhD, Greece Willibald Ruch, PhD, Switzerland Fons J.R. van de Vijver, PhD, The Netherlands

Volume 1, 2015, vi + 234 pages US $63.00 / € 44.95 ISBN 978-0-88937-437-9

www.hogrefe.com

Each volume in the series Psychological Assessment – Science and Practice presents the state-of-the-art of assessment in a particular domain of psychology, with regard to theory, research, and practical applications.

Editors and contributors are leading authorities in their respective fields. Each volume discusses, in a reader-friendly manner, critical issues and developments in assessment, as well as well-known and novel assessment tools.

The series is an ideal educational resource for researchers, teachers, and students of assessment, as well as practitioners.

Volume 2, 2016, vi + 346 pages US $69.00 / € 49.95 ISBN 978-0-88937-452-2

Volume 3, 2016, vi + 336 pages US $69.00 / € 49.95 ISBN 978-0-88937-449-2

How to assess the social atmosphere in forensic hospitals and identify ways of improving it “All clinicians and researchers who want to help make forensic treatment environments safe and effective should buy this book.”

Mary McMurran, PhD, Professor of Personality Disorder Research, Institute of Mental Health, University of Nottingham, UK

Norbert Schalast / Matthew Tonkin (Editors)

The Essen Climate Evaluation Schema – EssenCES A Manual and More

2016, x + 108 pp.

US $49.00 / € 34.95 ISBN 978-0-88937-481-2 Also available as eBook The Essen Climate Evaluation Schema (EssenCES) described here is a short, well-validated questionnaire that measures three essential facets of an institution’s social atmosphere. An overview of the EssenCES is followed by detailed advice on how to administer

www.hogrefe.com

and score it and how to interpret findings, as well as reference norms from various countries and types of institutions. The EssenCES “manual and more” is thus a highly useful tool for researchers, clinicians, and service managers working in forensic settings.

Multistudy Report

A Theoretical Approach to Resolving the Psychometric Problems Associated With the Zimbardo Time Perspective Inventory Results From the USA, Australia, Slovenia, and the United Kingdom Frank C. Worrell,1 Elizabeth C. Temple,2 Michael T. McKay,3 Urška Živkovič,4 John L. Perry,5 Zena R. Mello,6 Bojan Musil,4 and Jonathan C. Cole3 1 2

Cognition and Development, University of California, Berkeley, CA, USA School of Health Sciences and Psychology, Federation University, Ballarat, Australia

3

Department of Psychological Sciences, University of Liverpool, UK

4

Department of Psychology, University of Maribor, Slovenia

5

Department of Sport, Health and Exercise Science, University of Hull, UK

6

Department of Psychology, San Francisco State University, San Francisco, CA, USA Abstract: The Zimbardo Time Perspective Inventory (ZTPI; Zimbardo & Boyd, 1999) assesses five time-related constructs – Past Negative (PN), Past Positive (PP), Present Fatalistic (PF), Present Hedonistic (PF), and Future (F) – and is one of the most frequently used time measures in the extant literature. Versions of the ZTPI have been translated into a variety of languages, but the psychometric support for ZTPI scores remains contested. We examined the internal consistency, structural validity, and convergent validity of scores on a version of the ZTPI that consisted only of items that specifically referenced time constructs, the ZTPI-TP. Participants consisted of five samples of adolescents and adults from four countries: Australia (653 adults), Slovenia (425 adolescents and adults), the United Kingdom (913 adolescents; 455 adults), and the United States (815 adolescents). Structural validity analyses provided stronger support for ZTPI-TP scores than for ZTPI scores, and convergent validity evidence also provided support for ZTPI-TP scores. However, analyses revealed that the PF and PH factors were still problematic, especially with regard to factor coefficients and internal consistency estimates. We concluded that the ZTPI-TP can form the basis for a more robust version of the ZTPI. Keywords: time perspective, reliability, validity, Zimbardo Time Perspective Inventory, ZTPI

Time perspective is a multidimensional construct that assesses the influence that cognitions and affect about the past, present, and future have on individual functioning. To date, the Zimbardo Time Perspective Inventory (ZTPI; Zimbardo & Boyd, 1999) is one of the most widely used instruments to assess the construct. Indeed, the ZTPI has been translated into several languages, and there are validation studies in the literature on scores in English (Worrell & Mello, 2007), French (Apostolidis & Fieulaine, 2004), Spanish (Díaz-Morales, 2006), Swedish (Carelli, Wiberg, & Wiberg, 2011), and Portuguese (Milfont, Andrade, Belo, & Pessoa, 2008). Zimbardo and Boyd also reported strong concurrent validity evidence for ZTPI scores with a variety of constructs. Ó 2016 Hogrefe Publishing

Despite the frequent use of the ZTPI, there is mixed evidence regarding the psychometric properties of ZTPI scores. Although both Apostolidis and Fieulaine (2004) and Díaz-Morales (2006) reported support for the fivefactor structure based on exploratory factor analyses (EFAs) in samples of 419 university students and 756 adults, respectively, studies using confirmatory factor analyses (CFAs) have been less supportive. Worrell and Mello (2007) reported poor fit indices in a sample of 815 American adolescents (comparative fit index [CFI] = .636), Milfont et al. (2008) reported poor fit indices (goodness-of-fit index [GFI] = .74 and CFI = .70) in a sample of 247 Brazilian university students, and Carelli et al. (2011) European Journal of Psychological Assessment (2018), 34(1), 41–51 DOI: 10.1027/1015-5759/a000313

42

obtained a CFI of .63 in a sample of 419 Swedish adults.

Internal consistency estimates have also been mixed, with some authors reporting moderate to high alpha coefficients for ZTPI scores (e.g., Apostolidis & Fieulaine, 2004 [.65 α .84]; Carelli et al., 2011 [.70 α .79]) and others reporting lower estimates (e.g., Milfont et al., 2008 [.46 α .67]).

A number of attempts have been made to create shorter forms of the ZTPI. These include 15-item (McKay, Andretta, Magee, & Worrell, 2014; Zhang, Howell, & Bowerman, 2013), 25-item (Laghi, Baiocco, Liga, Guarino, & Baumgartnet, 2013), and 36-item (Sircova et al., 2014) versions, as well as an earlier version of the ZTPI, the 22-item Stanford Time Perspective Inventory (D’Alessio, Guarino, De Pascalis, & Zimbardo, 2003). However, scores on these shorter versions have also received mixed psychometric support in subsequent studies (McKay, Worrell, Temple, Perry, & Cole, 2014; McKay et al., 2015; Sircova et al., 2014).

Previous attempts to create a short form of the ZTPI have all used data-driven approaches (e.g., using the highest loading items or permitting correlated error terms to improve fit). These approaches are problematic insofar as adequate fit is then only sample-specific, and the findings do not replicate. For example, the factorial validity and reliability of a recent 15-item short-form of the ZTPI (the SZTPI-15; Zhang et al., 2013) were not well supported in a subsequent study (McKay, Worrell, et al., 2014). This problem is not specific to the ZTPI and appears to be common in time perception measures. For example, in their development of the Consideration of Future Consequences Scale-14, Joireman, Shaffer, Balliet, and Strathman (2012) allowed seven correlated error terms between items in order to achieve a good-fitting model for a hypothesized two-factor structure. However, a subsequent study failed to support the two-factor structure and had to permit an additional correlated error term in order to achieve good fit (Khachatryan, Joireman, & Casavant, 2013).

With specific reference to the ZTPI, some have previously argued that the scale is problematic because some items are measuring constructs such as fatalism, hedonism, and planning rather than time perspective (e.g., Crockett, Weinman, Hankins, & Marteau, 2009; Seijts, 1998; Shipp, Edwards, & Schurer-Lambert, 2009; Worrell & Mello, 2007; Worrell, Mello, & Buhl, 2013). For example, ZTPI Item 18, “It upsets me to be late for appointments,” is hypothesized to assess future orientation, although it could be argued that it in fact measures conscientiousness.

Similarly, several items hypothesized to measure present hedonism appear to measure impulsivity and risk-taking, such as “I do things impulsively” (#8) and “Taking risks keeps my life from becoming boring” (#31). Indeed, Worrell and Mello (2007) reported that the Future subscale split European Journal of Psychological Assessment (2018), 34(1), 41–51

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

into two factors, which they labeled Future and Future Planning. Similarly, in Crockett et al.’s examination of the 22-item version, they identified four factors – Future, Hedonism, Conscientiousness, and Present – two of which were not time-related constructs. However, the hypothesis that problems with the ZTPI may be related to the inclusion of items that do not measure time perspective remains untested.

Time perspective is considered an important psychological construct in adolescent and adult populations (Stolarski, Fieulaine, & van Beek, 2015), and the ZTPI has been put forward as the best way to measure the construct. For example, in a recent study examining ZTPI scores in samples of convenience from 24 countries, Sircova et al. (2014) concluded that the five ZTPI scores are (a) “valid and reliable index[es] of individual differences in time perspective” (p. 1) and (d) “the ‘gold standard’ for further research on time perspective” (p. 9). Not only are these claims in conflict with some of the evidence for ZTPI scores in the literature (e.g., Carelli et al., 2011; McKay, Worrell, et al., 2014, McKay et al., 2015), they are not supported by the results Sircova et al. reported. Sircova et al.’s invariance indices for ZTPI scores across countries (N = 10,765) were in the acceptable range for the root mean square error of approximation (RMSEA; .057) and standardized root mean square residual (SRMR; .062), but not for the CFI (.86). These discrepancies need to be examined empirically (American Educational Research Association, American Psychological Association and National Council on Measurement in Education, 2014).

The Current Study Given the failure of data-driven approaches in modifying the ZTPI and the criticisms about multiple constructs being measured on ZTPI subscales (Crockett et al., 2009; Shipp et al., 2009; Worrell et al., 2013), the present multisample study tested both the original five-factor, 56-item version of the scale, and a five-factor model consisting only of items that contained time-specific words or phrases (e.g., the past, tomorrow, on time, the moment). This study was also prompted in part by studies of a more recent instrument with items that focus only on time, the Adolescent Time Inventory-Time Attitudes (ATI-TA; Worrell & Mello, 2007). Fit indices for the structure of ATI-TA scores have been strong in Germany (N = 316; RMSEA = .033; SRMR = .050; CFI = .965; Worrell et al., 2013), New Zealand (N = 579; RMSEA = .041; SRMR = .055; CFI = .955; Alansari, Worrell, Rubie-Davies, & Webber, 2013), and the United States (N = 300; RMSEA = .037; SRMR = .059; CFI = .944; Worrell et al., 2013), and ATI-TA scores have demonstrated strong invariance between the US and Germany (RMSEA = .037; SRMR = .057; CFI = .949; Worrell et al., 2013). Ó 2016 Hogrefe Publishing

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

Thus, in this study of the ZTPI, all items that did not refer to time explicitly were eliminated. For example, items such as “I’ve made mistakes in the past that I wish I could undo” (#27) or “I try to live my life as fully as possible, one day at a time” (#17) were retained. However, items such as “I make lists of things to do” (#43) or “Often luck pays off better than hard work” (#53) were not included. We hypothesized that this time-based approach would yield a more theoretically-consistent version of the measure with psychometrically sound scores.

Our aim was to generate an instrument that assesses time perspective independent of other psychological constructs. Thus, we examined the structural validity of scores on the original ZTPI and the structural validity and internal consistency of scores on the temporal-phrasing ZTPI (the ZTPI-TP). We also investigated the convergent and discriminant validity of the ZTPI-TP with constructs used to validate the ZTPI, including consideration of future consequences, aggression, self-esteem, attachment, alcohol use, and mental health (Keough, Zimbardo, & Boyd, 1999; Zimbardo & Boyd 1999). The study included five samples from four countries – Australia, Slovenia, the United Kingdom, and the United States – and different developmental periods, although we present the results in combined tables for ease of comparison and interpretation.

Four of the samples were samples of convenience collected by time perspective researchers in the four countries, and one sample, the British adolescents, consisted of a sample of adolescents recruited to reflect the population.

The primary questions in this study were psychometric: Do time-specific ZTPI scores show evidence of internal consistency, structural validity, and convergent and discriminant validity? If time perspective is to be a useful cross-cultural construct for research and application (Stolarski et al., 2015), the scores need to be measured with integrity in each national context before they can be compared (International Test Commission, 2005). Thus, the five samples provided a robust examination of the cross-cultural validity claims being made for ZTPI scores (Sircova et al., 2014), and two of the countries in the current study – the US and the UK – were among those represented in Sircova et al.’s study. If the ZTPI is indeed the gold standard of time perspective measurement cross-culturally, the psychometric evidence should be supportive of the scores.

Method Participants Data from five independent samples – two from the United Kingdom (UK), and one each from the United States (US), Australia, and Slovenia – were analyzed. Participants in the Ó 2016 Hogrefe Publishing

43

first UK sample were 913 pupils (aged 11–16 years; 49.8% male) from High schools in Northern Ireland recruited by purposive sampling to reflect the overall demographics of the area. Participants in the second UK sample were 455 University undergraduates (aged 18–25 years; 49.7% male) recruited from Universities in Liverpool through opportunistic and snowball sampling. Participants in the US sample were 816 academically talented adolescents (aged 11–18 years; 46.6% male) attending a summer program at a research university in a Western state. Students were accepted into the summer program using several criteria, including school achievement, teacher recommendations, and an academic product. Participants were predominantly in the 7th–11th grades.

The 697 participants in the Australian sample were a general population sample (aged 17–70 years, M = 29.45; 67.8% female) recruited online through social media (e.g., Facebook, forums) via email snowballing, and through posts placed on university learning management systems (e.g., Moodle, Blackboard). Participants in the Slovenian sample were 425 adolescents and young adults (aged 15– 29 years, M = 20.80; 70.4% female) who completed an online questionnaire sent to them via email or social media (e.g., Facebook).

Measures Time Perspective The ZTPI (Zimbardo & Boyd, 1999) is a 56-item scale measuring the five factors of time perspective: Past Negative (PN), Past Positive (PP), Present Fatalistic (PF), Present Hedonistic (PH), and Future (F). PN reflects a negative or aversive view of the past and PP reflects a warm, sentimental attitude toward the past. Scores on PH reflect an orientation toward present pleasure with little concern for future consequences whereas PF describes a helpless and hopeless attitude toward the future and their life, and F indicates behavior dominated by striving for future goals and rewards. Responses were on a 5-point Likert-type scale from 1 (= very unlike me) to 5 (= very like me). The five-factor structure was supported using EFAs and confirmatory factor analyses (CFAs), and inter-subscale correlations were generally low (|.09| r |.38|). Convergent and discriminant validity were established based on meaningful correlations in the hypothesized directions with several constructs, including aggression, depression, self-esteem, and trait anxiety (Zimbardo & Boyd, 1999).

Twenty-five Items were retained from the original ZTPI (Table 2; Zimbardo & Boyd, 1999) in the ZTPI-TP: five PN items – 16, 22, 27, 36, and 50 (e.g., #16 – Painful past experiences keep being replayed in my mind); five PP items – 2, 7, 11, 15, and 25 (e.g., #7 – It gives me pleasure to think about my past; five PH items – 12, 17, 19, 28, and 46 European Journal of Psychological Assessment (2018), 34(1), 41–51

44

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

(e.g., #19 – Ideally, I would live each day as if it were my last); four PF items – 37, 39, 47, and 52 (e.g., #39 – It doesn’t make sense to worry about the future since there is nothing I can do about it anyway); and six F items – 6, 13, 21, 24, 40, and 56 (e.g., #13 – Meeting tomorrow’s deadlines and doing other necessary work comes before tonight’s play). All authors were in agreement about the items to be retained.

items are reverse-scored). Responses are given on a 4-point Likert-type scale from 1 (= strongly disagree) to 4 (= strongly agree). RSES scores have yielded strong reliability and validity coefficients across different samples with α coefficients ranging from 0.72 to 0.83 (Gray-Little, Williams, & Hancock, 1997; α in the current study = .82; e.g., I am able to do things as well as most other people).

Consideration of Future Consequences The Consideration of Future Consequences Scale (CFCS; Strathman, Gleicher, Boninger, & Edwards, 1994) is a 12-item scale made up of five positively worded items and seven negatively worded items. Responses were on a 5-point Likert-type scale from 1 (= very unlike me) to 5 (= very like me). As in other studies (e.g., Joireman, Balliet, Sprott, Spangenberg, & Schultz, 2008), in this study, the positively worded items were summed to yield a CFC-F (future) score (e.g., I am willing to sacrifice my immediate happiness or well-being in order to achieve future outcomes), indicating active consideration of future consequences. The negatively worded items were not reverse-scored and were summed to yield a CFC-I (immediate) score (e.g., I only act to satisfy immediate concerns, figuring the future will take care of itself), so that CFC-I scores reflect active consideration of immediate consequences or a present orientation.

Strathman et al. reported internal consistency estimates for CFSC scores in college student samples ranging from 0.80 to 0.86, a 2-week test-retest reliability coefficient of .76, and a 5-week test-retest reliability coefficient of .72 (α current study = .78 for CFC-F and .81 for CFC-I).

Relationships With Parents The Parents Scale of the Inventory of Parent and Peer Attachment-Revised (IPPA-R; Gullone & Robinson, 2005) was used to assess adolescents’ perceptions of the positive and negative affective and cognitive dimensions of relationships with their parents, particularly how well these figures serve as sources of psychological security. Responses are on a 5-point Likert-type scale from 1 (= almost never true) to 5 (= almost always true). The parental scale has 28 items that are distributed across three subscales: parental trust (10 items, α = .77; current α = .77; e.g., I trust my parents), parental communication (10 items, α = .77; current α = .84; e.g., I feel silly or ashamed when I talk about my problems with my parents), and parental alienation (8 items, α = .77; current α = .72; e.g., I don’t know who I can depend on).

Aggression The Aggression Questionnaire (AQ; Buss & Perry, 1992) consists of 29 items and assesses four constructs. The verbal aggression subscale has five items (α = .72; α in the current study = .68; e.g., I often find myself disagreeing with people), physical aggression consists of nine items (α = .85; current α = .89; e.g., If somebody hits me, I hit back), anger has seven items (α = .83; current α = .85; e.g., I have trouble controlling my temper), and hostility consists of eight items (α = .77; current α = .85; e.g., I am suspicious of overly friendly strangers). Respondents use a 5-point Likert-type scale from 1 (= very unlike me) to 5 (= very like me). AQ subscales have had their strongest correlations with impulsiveness, assertiveness, and competitiveness, with anger correlating most highly with impulsiveness. Buss and Perry also reported acceptable test-retest coefficients (.72 r .80). In the present study, subscale scores were summed to yield an overall aggression score (α = .79).

Self-Esteem The Rosenberg Self-Esteem Scale (RSES; Rosenberg, 1989) is a 10-item self-report measure of global self-esteem (five

European Journal of Psychological Assessment (2018), 34(1), 41–51

Psychological Well-Being in the UK The Hospital Anxiety and Depression Scale (HADS; Zigmond & Snaith, 1983) was used to measure the levels of anxious and depressive symptomatology in the UK. As the HADS assesses cognitive symptoms, it is not influenced by the physiological impact of the student lifestyle (e.g., alcohol consumption). The HADS produces scores for anxiety (HADS-A; e.g., I can sit at ease and feel relaxed) and depression (HADS-D; e.g., I have lost interest in my appearance) on separate subscales with scores ranging from zero to 28. Scores of 11 or more on either scale indicate clinically significant levels of depression and anxiety (Crawford et al., 2001; Snaith & Zigmond, 1994). Scores on both subscales of the HADS have been shown to be valid and internally consistent (HADS-A: α = .83; HADS-D: α = .82), with equal levels of sensitivity (.80) and specificity (.80; see Bjelland et al., 2002). Internal consistency estimates in the present study were as follows: HADS-A α = .84; HADS-D α = 88.

Alcohol Use The Alcohol Use Disorders Identification Test (AUDIT; Saunders, Aasland, Babor, de la Fuente, & Grant, 1993) was used to assess problematic alcohol use. AUDIT is a 10-item questionnaire (e.g., How often do you have a drink containing alcohol?) with valid and reliable scores across different contexts and cultures (e.g., de Meneses-Gaya, Zuardi, Loureiro, & Crippa, 2009; Reinert & Allen, 2007).

When used to detect problematic alcohol use in a population of university undergraduates, AUDIT

Ó 2016 Hogrefe Publishing

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

demonstrated good sensitivity (.94) and specificity (.92; Adewuya, 2005). The reliability estimate for AUDIT scores in the present study was .83.

Psychological Well-Being in Australia The 18-item version of the Mental Health Inventory (MHI18; McHorney, Ware, Rogers, Raczek, & Lu, 1992; Ware, Manning, Duan, Wells, & Newhouse, 1984) was used to assess psychological well-being in Australia. Participants responded to statements across four subscales to indicate how often they have felt certain ways during the past four weeks using a 6-point scale (1 = all of the time, 6 = none of the time): depression (MHI-D, four items; e.g., Did you feel depressed?), anxiety (MHI-A, 5 items; e.g., Have you been anxious or worried?), loss of behavioral/emotional control (MHI-B, 4 items; e.g., Have you been in firm control of your behaviour, thoughts, emotions, feelings?), and positive affect (MHI-P, 4 items; e.g., Have you felt cheerful, light-hearted?). Subscale raw scores are converted to index scores with a range of 0–100, where higher scores are indicative of higher levels of the psychological construct (e.g., a higher MHI-D score indicates higher levels of depression symptomology and a higher MHI-P score indicates higher levels of positive affect). MHI-18 scores have been found to have good reliability and validity (McHorney et al., 1992; Ware et al., 1984), and the reliability estimates of subscale scores in the present study were good: MHI-D α = .91; MHI-A α = .86; MHI-B α = .87; and MHI-P α = .86.

The revised version of the Experiences in Close Relationships Questionnaire (ECR-R; Fraley, Waller, & Brennan, 2000) was used to assess adult attachment orientation.

Participants responded to 36 items on a 7-point scale (1 = strongly disagree, 7 = strongly agree), indicating how they felt about someone with whom they were in a relationship.

Two subscale scores are calculated – attachment anxiety (ECR-Ax, 18 items; e.g., I often worry that my partner doesn’t really love me) and attachment avoidance (ECRAv, 18 items; e.g., I find it difficult to allow myself to depend on romantic partners) – both with scores ranging from 0 to 7, where higher scores indicate higher levels of attachment insecurity. ECR-R scores have been shown to have good levels of internal consistency (Fraley et al., 2000), and the reliability of the subscale scores in the present study was high (ECR-Ax α = .95; ECR-Av α = .94).

Procedure and Statistical Analyses Three of the four countries in this study were Englishspeaking and the studies in those countries used the original ZTPI items. In Slovenia, the CFCS and ZTPI were translated into Slovenian using recommended practices (Geisinger, 2003). In addition to translating items into Slovenian and back-translating them into English, the Ó 2016 Hogrefe Publishing

45

translated items were examined to assure that the Slovenian items retained the same meaning as in English (International Test Commission, 2005). This process indicated that the Slovenian translations were satisfactory.

The ZTPI was included in every sample. Other measures were administered as follows. In the UK adolescent sample, self-esteem was included in eight schools (n = 735; 81%), aggression in three schools (n = 333; 36%), and parental attachment in two schools (n = 133; 15%). The Australian study included problematic alcohol use (n = 371; 53%), psychological well-being (n = 213; 31%), and adult attachment orientation (n = 463; 66%), and the Slovenian study included consideration of future consequences. Finally, the UK university study included an assessment of anxiety and depression. Participants in the two UK samples completed paper-and-pencil surveys in their home classrooms, and participants in the US samples completed paperand-pencil surveys on their own time and returned them to the classroom. The Australian and Slovenian participants completed the survey online.

Model fit was assessed using both CFAs and exploratory structural equation modeling (ESEM) in Mplus 7 (Muthén & Muthén, 1998–2012) with the MLM estimator to account for departure from multivariate normality. ESEM (see Asparouhov & Muthén, 2009) integrates confirmatory and exploratory factor analysis (Weisner & Schandling, 2013). However, ESEM differs from standard CFA in that all factor loadings are estimated, while observing various constraints necessary for model identification, and factor loading matrices can be rotated. Marsh et al. (2009) have argued that ESEM is a viable alternative to CFA for psychological scales composed of indicators with many nonzero cross-loadings.

Also referred to as the Satorra-Bentler w2, the MLM maximum likelihood parameter estimates with standard errors and a mean-adjusted chi-square (w2) test statistic that is robust to non-normality. We assessed a five-factor model for the ZTPI (56 items) and the ZTPI-TP (25 items). An oblique geomin rotation, as recommended by Marsh et al.

(2009), with an epsilon value of 0.5 and ML estimation were used in all ESEM analyses. These options are recommended when there are more than four response categories (Beauducel & Herzberg, 2006) and data may not be normally distributed (Bentler & Wu, 2002).

The indices used to test model fit were w2, CFI, the Tucker-Lewis index (TLI), RMSEA, and SRMR. Although Hu and Bentler (1999) have recommended stringent cut-offs for fit indices (i.e., > .95 for CFI and TLI, < .06 for RMSEA, and < .08 for SRMR), Perry and colleagues (2013) suggested that strict adherence to these cut-off values often leads to erroneous results, as factor loadings in social sciences are typically lower (see, e.g., Heene, Hilbert, Draxler, Ziegler, & Bühner, 2011; Marsh, Hau, & Wen, 2004). We also European Journal of Psychological Assessment (2018), 34(1), 41–51

46

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

Table 1. CFA and ESEM fit indices for ZTPI scores

CFA American sample Australian sample British adolescent British University Slovenian sample ESEM American sample Australian sample British adolescent British University Slovenian sample

w2s-b

df

CFI

TLI

SRMR

4,484.84* 4,297.11* 4,517.21* 3,629.35* 3,789.61*

1,474 1,474 1,474 1,474 1,474

.65 .61 .71 .66 .65

.63 .59 .70 .65 .63

.08 .08 .07 .08 .10

.050 .058 .048 .057 .060

(.048, (.055, (.046, (.054, (.058,

.052) .059) .049) .059) .063)

3,100.73* 2,843.40* 3,068.98* 2,832.84* 2,678.21*

1,270 1,270 1,270 1,270 1,270

.82 .81 .85 .78 .82

.79 .77 .82 .73 .78

.04 .04 .04 .04 .04

.041 .047 .040 .052 .050

(.040, (.044, (.038, (.049, (.048,

.044) .048) .041) .055) .054)

RMSEA (90% CI)

Notes. ZTPI = Zimbardo Time Perspective Inventory; CFA = Confirmatory Factor Analysis; ESEM = Exploratory Structural Equation Modeling. *p < .05.

Table 2. Number of standardized estimates > .50 for CFA and ESEM analyses for ZTPI scores American (n = 815)

PN (10 items) PP (9 items) PF (9 items) PH (15 items) F (13 items)

Australian (n = 653)

British adolescent (n = 913)

British University (n = 455)

Slovenian (n = 425)

CFA

ESEM

CFA

ESEM

CFA

ESEM

CFA

ESEM

CFA

ESEM

5 4 3 2 4

4 3 3 2 3

6 4 3 4 3

0 3 0 3 3

4 5 3 4 6

4 5 1 2 4

7 4 4 7 7

7 4 4 4 4

9 4 3 8 7

4 4 3 2 6

Notes. ZTPI = Zimbardo Time Perspective Inventory; PN = Past Negative; PP = Past Positive; PF = Present Fatalistic; PH = Present Hedonistic; F = Future.

examined standardized parameter estimates. Factor loadings for CFA were interpreted using Comrey and Lee’s (1992) recommendations (i.e., > .71 = excellent, > .63 = very good, > .55 = good, > .45 = fair, and > .32 = poor). Additionally, Spearman’s ρ correlations were computed between time perspective measures and other psychosocial measures.

These latter analyses were performed in SPSS V.20.

Results Structural Validity Analyses Table 1 displays the results of both CFA and ESEM model fit analyses for scores on the 56-item ZTPI. Results of the CFAs yielded mixed results. The SRMR and RMSEA values in the ESEM analyses were good; however, the CFI and TLI values were well below the minimally acceptable (> .90) threshold. Table 2 summarizes the numbers of items among the original 56 which loaded > .50 in all five samples in both analyses. Fewer than half of the 56 items met this criterion, with percentages doing so in each sample as follows for CFAs/ESEMs: British Adolescent = 39.3/ 28.6%; American = 32.1/26.8%; Australian = 35.7/16.0%; Slovenian = 55.4/33.9%; British University = 41.1/33.9%.

Table 3 displays the results of both CFA and ESEM model fit analyses for scores on the 25-item ZTPI-TP. European Journal of Psychological Assessment (2018), 34(1), 41–51

Results of the CFAs revealed poor fit for the data. However, results for the ESEM models were all good (< .08/.05) for the SRMR and RMSEA values, and acceptable (> .90) to good (> .95) for the CFI values. Moreover, in these models, substantive cross-loadings (i.e., > .30) were rare. The British Adolescent sample did not have any cross-loadings. The American, and Australian, and British University samples yielded just two cross-loadings each, and the Slovenian sample had three. All samples converged with the exception of the British University sample in ESEM analysis, in which Item 39 created a linear dependency. Consequently, this item was removed from this particular model to enable convergence and model estimation. Table 4 summarizes the numbers of items which loaded > .50 in all five samples in both analyses. As can be seen, on only one occasion (CFA in Slovenian sample) did all items load > .50 on their hypothesized factor. Loadings for the PH and PF factors were particularly problematic.

Running invariance analyses on models that fail to achieve satisfactory model fit is counterintuitive, as the increasingly constrained models will further harm model fit. However, to examine the relative strengths of the original and temporal-phrasing scores across national contexts, we conducted invariance testing on the ZTPI and the ZTPI-TP. Invariance was assessed across all five samples (N = 3,261). First, configural invariance was assessed by replicating the CFA-ICM (independent cluster model) across all samples. Second, factors were constrained to test Ó 2016 Hogrefe Publishing

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

47

Table 3. CFA and ESEM fit indices for ZTPI-TP scores

CFA American sample Australian sample British adolescent British University Slovenian sample ESEM American sample Australian sample British adolescent British University Slovenian sample

w2s-b

df

CFI

TLI

SRMR

1,304.80* 912.76* 1,157.40* 730.00* 929.53*

265 265 265 265 265

.64 .77 .71 .76 .69

.60 .74 .67 .73 .65

.08 .07 .07 .07 .11

.069 .062 .060 .062 .078

(.066, (.057, (.057, (.057, (.071,

.073) .066) .064) .067) .082)

185 185 185 166a 185

.91 .95 .95 .91 .94

.86 .93 .92 .86 .90

.03 .03 .02 .03 .03

.047 .038 .035 .049 .044

(.041, (.030, (.028, (.042, (.037,

.050) .042) .038) .056) .052)

496.63* 339.03* 368.35* 347.26* 340.42*

RMSEA (90% CI)

Notes. ZTPI-TP = Zimbardo Time Perspective Inventory-Temporal Phrasing; CFA = Confirmatory Factor Analysis; ESEM = Exploratory Structural Equation Modeling. a Item 29 created a linear dependency in this model and was removed to enable convergence. *p < .05.

Table 4. Number of standardized estimates > .50 for CFA and ESEM analyses for ZPTI-TP scores American (n = 815)

PN (5 items) PP (5 items) PF (4 items) PH (5 items) F (6 items)

Australian (n = 653)

British adolescent (n = 913)

British University (n = 455)

Slovenian (n = 425)

CFA

ESEM

CFA

ESEM

CFA

ESEM

CFA

ESEM

CFA

ESEM

3 2 0 0 3

2 3 0 0 2

3 3 1 1 2

4 2 1 1 3

3 2 0 1 3

3 2 0 0 3

3 3 3 4 2

3 2 1 0 2

3 5 3 1 3

2 3 1 2 3

Notes. ZTPI-TP = Zimbardo Time Perspective Inventory-Temporal Phrasing; PN = Past Negative; PP = Past Positive; PF = Present Fatalistic; PH = Present Hedonistic; F = Future.

Table 5. Invariance analyses of ZTPI and ZTPI-TP scores Model 56-Item ZTPI Configural invariance Metric invariance Scalar invariance Residual invariance 25-Item ZTPI-TP Configural invariance Metric invariance Scalar invariance Residual invariance

w2

df

Δw2

Δdf

CFI

TLI

SRMR

20,539.93* 21,525.91* 26,511.23* 27,323.07*

7,370 7,574 7,778 7,798

– 985.98 4,985.32 811.84

– 204 204 20

.674 .655 .537 .517

.660 .649 .542 .523

.080 .085 .091 .104

.052 .053 .061 .062

(.052, (.052, (.060, (.061,

.053) .054) .062) .063)

5,053.77* 5,433.40* 6,916.61* 7,699.26*

1,325 1,405 1,485 1,505

– 379.63 1,483.21 782.65

– 80 80 20

.712 .689 .581 .522

.674 .668 .577 .524

.080 .084 .092 .110

.066 .066 .075 .079

(.064, (.064, (.073, (.078,

.068) .068) .077) .081)

RMSEA (90% CI)

Notes. ZTPI = Zimbardo Time Perspective Inventory; ZTPI-TP = Zimbardo Time Perspective Inventory-Temporal Phrasing. *p < .001.

metric invariance. Third, we examined scalar invariance by constraining factors and item intercepts. Finally, residual variance was tested by factors, item intercepts, and factor means. Results are displayed in Table 5. As can be seen, CFI and TLI indices were poor, the SRMR and RMSEA were marginal, providing no support for configural invariance. Model invariance is supported by little or no change in model fit on the increasingly constrained models.

Cheung and Rensvold (2002) suggested ΔCFI .01, although Meade, Johnson and Braddy (2008) recommended a much stricter criterion of ΔCFI .002 to support invariance. Even by the more liberal criterion proposed by Ó 2016 Hogrefe Publishing

Cheung and Rensvold, ZTPI and ZTPI-TP scores did not achieve even weak invariance across the samples.

Internal Consistency for ZTPI-TP Scores Table 6 displays the results of internal reliability (α and ω) estimates for scores on the five factors of the ZTPI-TP. Overall, these reliability estimates were modest to moderate and generally similar. Employing a value of .60 as acceptable, only 12 of the 25 alpha coefficients and 15 of the omega coefficients were acceptable and only PN scores European Journal of Psychological Assessment (2018), 34(1), 41–51

48

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

Table 6. Alpha (α) and Omega (ω)a estimates for ZTPI-TP scores PN α British adolescent American Australian Slovenia British University Median values

.63 .72 .76 .78 .73 .73

PP ω .64 .72 .77 .79 .74 .77

α

PF ω

.63 .57 .72 .66 .48 .63

.67 .60 .73 .68 .69 .68

α .26 .44 .46 .50 .38 .44

PH ω

α

.31 .44 .47 .51 .58 .47

.50 .44 .45 .61 .48 .48

F ω

α

.47 .43 .43 .60 .42 .43

ω

.67 .63 .63 .55 .47 .63

.67 .63 .63 .54 .64 .63

Notes. ZTPI-TP = Zimbardo Time Perspective Inventory-Temporal Phrasing; PN = Past negative; PP = Past Positive; PF = Present Fatalistic; PH = Present Hedonistic; F = Future. a Omega estimates are based on the coefficients from the confirmatory factor analyses (CFAs).

Table 7. Concurrent validity correlations (p < .01) for ZTPI-TP scores

Slovenia Con. of future consequences-immediate Con. of future consequences-future British University HADS-Anxiety HADS-Depression AUDIT-Alcohol use disorders Australia MHI-Anxiety MHI-Depression MHI-Behavioral/emotional control MHI-Positive affect AUDIT-Alcohol use disorders ECC-R-Attachment anxiety ECC-R-Avoidance anxiety British adolescent Parental attachment Self-Esteem Aggression

Past negative

Past positive

Present fatalistic

Present hedonistic

– –

– –

.42 –

.25 –

.44 .37

.33 .32 –

– – –

– .22 .20

– – .24

– .26 .28

.57 .57 .53 .54 .19 .44 .27

.44 .37 .42 .48 – .20 .20

.30 .31 .33 .32 .15 .29 .20

– – – .25 – .13 –

– – – – .18 – –

.34 .35 .38

– – .28

.25 – .31

– – –

Future

.33 – .32

Notes. ZTPI-TP = Zimbardo Time Perspective Inventory-Temporal Phrasing; Con = Consideration; HADS = Hospital Anxiety and Depression Scale; AUDIT = Alcohol Use Disorders Identification Test; MHI = Mental Health Inventory; ECC-R = Revised Experiences in Close Relationships Questionnaire.

showed evidence of acceptable internal consistency across the five samples. Internal consistency estimates were acceptable in three or four samples for PP and F scores, and in one sample for PH scores. PF scores, which were based on only four items, did not achieve an acceptable internal consistency in any sample (see Table 6).

Concurrent Validity for ZTPI-TP Scores In the final set of analyses, we examined the relationship of ZTPI-TP subscale scores to other psychological constructs.

Table 7 includes all correlations that were statistically significant at the .01 level, although we only interpret correlations that had medium effect sizes (i.e., .30). In the Slovenian sample, PF scores were positively related to immediate consequences and F scores were positively related to future consequences and negatively related to immediate consequences. In the British University sample, PN scores had positive relationships with anxiety European Journal of Psychological Assessment (2018), 34(1), 41–51

and depression scores. The other significant correlations had theoretically congruent relationships but were less than .30.

In the Australian sample, PN and PF had meaningful positive correlations with anxiety, depression, and behavioral/emotional control, and meaningful negative correlations with positive affect, although the PN correlations were substantially higher (> .20) than the PF correlations.

PP scores were meaningfully related to these same four constructs, but positively related to positive affect and negatively related to anxiety, depression, and behavioral/ emotional control. The only other meaningful correlation in the Australian sample was a positive relationship between PN scores and attachment anxiety. Finally, in the British Adolescent sample, PN was negatively related to parental attachment and self-esteem, but positively related to aggression. PF was positively related to aggression, and F was positively related to parental attachment and negatively related to aggression. PH did not have meaningful associations with any measures. Ó 2016 Hogrefe Publishing

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

Discussion A growing body of research has raised questions about the psychometric properties of ZTPI scores (e.g., Crockett et al., 2009; Shipp et al., 2009; Worrell & Mello, 2007), with authors reporting a variety of factor solutions and problems with particular items. In response, a number of attempts have been made to create shortened forms of the scale through data-driven methods (e.g., Sircova et al., 2014; Zhang et al., 2013). However, these short forms have generally not been supported in subsequent analyses (e.g., McKay, Worrell, et al., 2014; McKay et al., 2015). In the present study, we employed a theoretical rationale based on previous criticisms of ZTPI item content and tested a shortened form of the scale – the ZTPI-TP – which contained 25 items that included only temporal phrasing. The totality of results suggests that this version of the ZTPI might be considered an answer, but not the definitive answer to the psychometric and theoretical issues within the ZTPI.

Psychometric Properties of ZPTI and ZTPI-TP Scores First of all, using CFA, results of factorial validity for the full 56-item version of the scale were suboptimal with the highest CFI and TLI values falling well short of the threshold of acceptable fit. The CFI and TLI indices improved with the use of ESEM, although the values were all still suboptimal.

Factorial validity results for the ZTPI-TP using CFA were also poor. However, with the use of ESEM, fit indices for ZTPI-TP scores in all five samples were adequate to good. Additionally, the concurrent validity of this shortened temporalphrasing version could be described as moderate to good, with ZTPI-TP scores having correlations in the expected directions with constructs correlated with ZTPI scores (see Zimbardo & Boyd, 1999) across four of the five samples.

These findings suggest that there are at least two problems with ZTPI scores. First, they are measuring multiple constructs as critics have suggested, and second, their variance is distributed across too many constructs. Removing the nontemporal items solves the first problem, but not the second, which is why the ESEM fit indices are so much better than CFA fit indices for ZTPI-TP scores, but not for ZTPI scores.

A third concern is internal consistency. The reliability coefficients for ZTPI-TP scores differed substantially by factor. Even employing a low minimum threshold for internal consistency, the results did not provide support for the PH and PF factors and provided modest support for the PP and F factors. In only one case, PN, there was an α value greater than .60 across all five samples, with alphas for scores on four of the five samples > .70. Elsewhere, Streiner (2003) highlighted (a) that Cronbach’s α values are directly influenced by the number of items used (such that it is Ó 2016 Hogrefe Publishing

49

more difficult to achieve a higher α with fewer items), and (b) that higher and lower Cronbach’s α values can be the result of the topic under investigation. Taking the example of the ZTPI-TP, and the breadth of its item content, a person might endorse PN16, “Painful past experiences keep being replayed in my mind” (indicating having suffered negative experiences), but not PN27, “I’ve made mistakes in the past that I wish I could undo” (indicating that they have not, in their view, made regrettable mistakes). However, omega estimates, which are based on factor coefficients, also reveal concern with the internal consistency of ZTPI-TP subscale scores.

One issue is that ZTPI items tap a broad range of behaviors, attitudes, and cognitions. Nonetheless, items are attempting to capture the same construct on a single factor need to meet a minimum threshold of internal consistency to allow for replication of results (Thompson, 2003) and for interpreting inferences based on the scores (American Educational Research Association et al., 2014). Moreover, four to six items on a well-constructed scale can produce alpha estimates in the .70–.90 range (e.g., the constructs used for concurrent validity in this study).

Conclusion and Future Research As in much of the literature on the ZTPI, four of the samples in this study were samples of convenience and instruments were administered in different ways. Although these differences might be limitations in another study, given the broad claims being made for ZTPI scores (Sircova et al., 2014), this diversity of samples and approaches is a strength in the current study. Given the results of this study, we are not in a position to recommend use of the ZTPI-TP as yet another shortened version of the ZTPI. Rather, we suggest using the ZTPI-TP items alone as the basis for the revision of the ZTPI, given its existing widespread use, and that consideration be given to the addition of new items that complement the more robust current ones. A longer scale consisting of carefully-selected, time-focused items may yield more acceptable CFA results and stronger internal consistency estimates. The popularity of the ZTPI in the time perspective literature highlights the need for scales that assess all three time periods, and the interest in time perspective is likely to grow, given recent studies highlighting its utility and predictive validity with a variety of constructs (e.g., Joireman et al., 2012; McKay, Andretta, et al., 2014).

Shipp et al. (2009) described time perspective as an overarching construct within which other more narrow constructs existed, and Lasane and O’Donnell (2005) underlined the fact that the construct has cognitive, affective, and behavioral dimensions (cf. Zimbardo & Boyd 1999).

However, it may be that time perspective is so broad that European Journal of Psychological Assessment (2018), 34(1), 41–51

50

the singular measurement of it is too ambitious. Thus, the creation of latent variables based on scores on more specific aspects of time perspective (e.g., temporal depth, time attitudes, consideration of future consequences, temporal focus, possible selves) may be a more robust way to assess this more overarching construct.

References Adewuya, A. O. (2005). Validation of the Alcohol Use Disorders Identification Test (Audit) as a screening tool for alcoholrelated problems among Nigerian university students. Alcohol and Alcoholism, 40, 575–577. doi: 10.1093/alcalc/agh197 Alansari, M., Worrell, F. C., Rubie-Davies, C., & Webber, M. (2013).

Adolescent Time Attitude Scale (ATAS) scores and academic outcomes in secondary school females in New Zealand.

International Journal of Quantitative Research in Education, 1, 251–274. doi: 10.1504/IJQRE.2013.057687 American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Apostolidis, T., & Fieulaine, N. (2004). Validation française de l’échelle de temporalité the Zimbardo Time Perspective Inventory (ZTPI) [French validation of the temporal scale the Zimbardo Time Perspective Inventory (ZTPI)]. Revue Européenne de Psychologie Appliquée, 54, 207–217.

Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397–438. doi: 10.1080/10705510903008204 Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186–203. doi: 10.1207/ s15328007sem1302_2 Bentler, P. M., & Wu, E. J. C. (2002). EQS 6 for Windows: User's guide. Encino, CA: Multivariate Software.

Bjelland, I., Dahl, A. A., Haug, T. T., & Neckelmann, D. (2002). The validity of the Hospital Anxiety and Depression Scale: An updated literature review. Journal of Psychosomatic Research, 52, 69–77. doi: 10.1016/S0022-3999(01)00296-3 Buss, A. H., & Perry, M. (1992). The Aggression Questionnaire.

Journal of Personality and Social Psychology, 63, 452–459. doi: 10.1037/0022-3514.63.3.452 Carelli, M. G., Wiberg, B., & Wiberg, M. (2011). Development and validation of the Swedish Zimbardo Time Perspective Inventory.

European Journal of Psychological Assessment, 27, 220–227. doi: 10.1027/1015-5759/a000076 Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-offit indexes for measurement invariance. Structural Equation Modeling, 9, 233–255. doi: 10.1207/S15328007SEM0902_5 Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis.

Hillsdale, NJ: Erlbaum.

Crawford, J. R., Henry, J. D., Crombie, C., & Taylor, E. P. (2001).

Normative data for the HADS from a large non-clinical sample.

British Journal of Clinical Psychology, 40, 429–434. doi: 10.1348/014466501163904 Crockett, R. A., Weinman, J., Hankins, M., & Marteau, T. (2009).

Time orientation and health-related behaviour: Measurement in general population samples. Psychology and Health, 24, 333–350. doi: 10.1080/08870440701813030

European Journal of Psychological Assessment (2018), 34(1), 41–51

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

D’Alessio, M., Guarino, A., De Pascalis, V., & Zimbardo, P. G. (2003). Testing Zimbardo’s Stanford Time Perspective Inventory (SPTI) – Short Form: An Italian study. Time and Society, 12, 333–347. de Meneses-Gaya, C., Zuardi, A.W., Loureiro, S. R., & Crippa, J. A. S. (2009). Alcohol Use Disorders Identification Test (AUDIT): An updated systematic review of psychometric properties.

Psychology and Neuroscience, 2, 83–97. doi: 10.3922/j. psns.2009.1.12 Díaz-Morales, J. F. (2006). Estructura factorial y fiabilidad del Inventario de Perspectiva Temporal de Zimbardo [Factorial Structure and Reliability of the Zimbardo Time Perspective Inventory]. Psicothema, 18, 565–571.

Fraley, R. C., Waller, N. G., & Brennan, K. A. (2000). An item response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology, 78, 350–365. doi: 10.1037/0022-3514.78.2.350 Geisinger, K. F. (2003). Testing and assessment in cross-cultural psychology. In V. J. R. Graham, J. A. Naglieri, & I. B. Weiner (Eds.), Handbook of psychology – Vol. 10: Assessment psychology (pp. 95–119). NJ: Wiley.

Gray-Little, B., Williams, V. S. L., & Hancock, T. D. (1997). An item response theory analysis of the Rosenberg Self-Esteem Scale.

Personality and Social Psychology Bulletin, 23, 443–451. doi: 10.1177/0146167297235001 Gullone, E., & Robinson, K. (2005). The Inventory of Parent and Peer Attachment-Revised (IPPA-R) for children: A psychometric investigation. Clinical Psychology & Psychotherapy, 12, 67–79. doi: 10.1002/cpp.433 Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011).

Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16, 319–336. doi: 10.1037/a0024917 Hu, L., & Bentler, P. M. (1999). Cut-off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modelling, 6, 1–55. doi: 10.1080/10705519909540118 International Test Commission. (2005). International guidelines on test and adaptation. Retrieved from www.intestcom.org Joireman, J., Balliet, D., Sprott, D., Spangenberg, E., & Schultz, J. (2008). Consideration of future consequences, ego-depletion, and self-control: Support for distinguishing between CFC-immediate and CFC-future sub-scales. Personality and Individual Differences, 45, 15–21. doi: 10.1016/j. paid.2008.02.011 Joireman, J., Shaffer, M. J., Balliet, D., & Strathman, A. (2012).

Promotion orientation explains why future-oriented people exercise and eat healthy: Evidence from the two-factor Consideration of Future Consequences-14 Scale. Personality and Social Psychology Bulletin, 38, 1272–1287. doi: 10.1177/ 0146167212449362 Keough, K. A., Zimbardo, P. G., & Boyd, J. N. (1999). Who’s smoking, drinking, and using drugs? Time perspective as a predictor of substance use. Basic and Applied Psychology, 21, 149–164. doi: 10.1207/S15324834BA210207 Khachatryan, H., Joireman, J., & Casavant, K. (2013). Relating values and consideration of future and immediate consequences to consumer preference for biofuels: A threedimensional social dilemma analysis. Journal of Environmental Psychology, 34, 97–108. doi: 10.1016/j.jenvp.2013.01.001 Laghi, F., Baiocco, R., Liga, F., Guarino, A., & Baumgartnet, E. (2013). Identity status differences among Italian adolescents: Associations with time perspective. Children and Youth Services Review, 35, 482–487.

Ó 2016 Hogrefe Publishing

F. C. Worrell et al., A Temporal-Item-Only Version of the ZTPI

Lasane, T. P., & O’Donnell, D. A. (2005). Time orientation measurement: A conceptual approach. In A. Strathman & J. Joireman (Eds.), Understanding behavior in the context of time: Theory, research, & application (pp. 11–30). Hillsdale, NJ: Erlbaum.

Marsh, H. W., Hau, K., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11, 320–341. doi: 10.1207/ s15328007sem1103_2 Marsh, H. W., Muthén, B., Asparouhov, A., Lüdtke, O., Robitzsch, A., Morin, A. J. S., & Trautwein, U. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students’ evaluations of university teaching. Structural Equation Modeling, 16, 439–476. doi: 10.1080/10705510903008220 McHorney, C. A., Ware, J. E., Rogers, W., Raczek, A. E., & Lu, J. F. R. (1992). The validity and relative precision of MOS short and long form Health Status Scales and Dartmouth COOP charts: Results from the medical outcomes study. Medical Care, 30, 253–265.

McKay, M. T., Andretta, J. R., Magee, J., & Worrell, F. C. (2014).

What do temporal profiles tell us about adolescent alcohol use?

Results from a large sample in the United Kingdom. Journal of Adolescence, 37, 1319–1328. doi: 10.1016/j.adolescence.

2014.09.008 McKay, M. T., Worrell, F. C., Temple, E. C., Perry, J., & Cole, J. C. (2014). A critical examination of the reliability and validity of the SZTPI-15 in British, American and Australian samples. Personality and Individual Differences, 69, 168–172. doi: 10.1016/j. paid.2014.05.030 McKay, M. T., Worrell, F. C., Temple, E. C., Perry, J., Cole, J. C., & Mello, Z. R. (2015). Less is not always more: The case of the 36item short form of the Zimbardo Time Perspective Inventory.

Personality and Individual Differences, 72, 68–71. doi: 10.1016/ j.paid.2014.08.018 Meade, A. W., Johnson, E. C., & Braddy, P. W. (2008). Power and sensitivity of alternate fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568–592. doi: 10.1037/0021-9010.93.3.568 Milfont, T. L., Andrade, P. R., Belo, R. P., & Pessoa, V. S. (2008).

Testing Zimbardo Time Perspective Inventory in a Brazilian sample. Revista Interamericana de Psicologia, 42, 49–58.

Muthén, B. O., & Muthén, L. K. (1998–2012). Mplus User’s Guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

Perry, J. L., Clough, P. J., Crust, L., Earle, K., & Nicholls, A. R. (2013). Factorial validity of the Mental Toughness Questionnaire-48. Personality and Individual Differences, 54, 587–592. doi: 10.1016/j.paid.2012.11.020 Reinert, D. F., & Allen, J. P. (2007). The Alcohol Use Disorders Identification Test: An update of research findings. Alcoholism: Clinical and Experimental Research, 31, 185–199. doi: 10.1111/ j.1530-0277.2006.00295.x Rosenberg, M. (1989). Society and the adolescent self-image (Rev. ed.). Middletown, CT: Wesleyan University Press.

Saunders, J. B., Aasland, O. G., Babor, T. F., de la Fuente, J. R., & Grant, M. (1993). Development of the Alcohol Use disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption-II. Addiction, 88, 791–804. doi: 10.1111/j.1360-0443.1993.tb02093.x Seijts, G. H. (1998). The importance of future time perspective in theories of work motivation. Journal of Psychology, 132, 154–168. doi: 10.1080/00223989809599156 Shipp, A. J., Edwards, J. R., & Schurer-Lambert, L. (2009).

Conceptualization and measurement of temporal focus: The subjective experience of the past, present, and future.

Ó 2016 Hogrefe Publishing

51

Organizational Behavior and Human Decision Processes, 110, 1–22. doi: 10.1016/j.obhdp.2009.05.001 Sircova, A., van de Vijver, F. J. R., Osin, E., Milfont, T. L., Fieulaine, N., Kislali-Erginbilic, A., & . . . 54 members of the International Time Perspective Research Project. (2014). A global look at time: A 24country study of equivalence of the Zimbardo Time Perspective Inventory. SAGE Open, 4, 1–12. doi: 10.1177/2158244013515686 Snaith, R. P., Zigmond, A. S. (1994). HADS: Hospital Anxiety and Depression Scale. Windsor, Canada: NFER Nelson.

Stolarski, M., Fieulaine, N., & van Beek, W. (Eds.). (2015). Time perspective theory: Review, research, and application. Essays in honor of Phillip G. Zimbardo. Zug, Switzerland: Springer.

Strathman, A., Gleicher, F., Boninger, D. S., & Edwards, C. S. (1994). The consideration of future consequences: Weighing immediate and distant outcomes of behavior. Journal of Personality and Social Psychology, 66, 742–752. doi: 10.1037/ 0022-3514.66.4.742 Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99–103. doi: 10.1207/ S15327752JPA8001_18 Thompson, B. (2003). Understanding reliability and coefficient alpha, really. In B. Thompson (Ed.), Score reliability: Contemporary thinking on reliability issues (pp. 3–23). Thousand Oaks, CA: Sage.

Ware, J. E. Jr., Manning, W. G. Jr., Duan, N., Wells, K. B., & Newhouse, J. P. (1984). Health status and the use of outpatient mental health services. American Psychologist, 39, 1090–1100. doi: 10.1037/0003-066X.39.10.1090 Weisner, M., & Schandling, G. T. (2013). Exploratory structural equation modeling, bifactor models, and standard confirmatory factor analysis models: Application to the BASC-2 Behavioral and Emotional Screening System Teacher Form. Journal of School Psychology, 51, 751–763. doi: 10.1016/j.jsp.2013.09.001 Worrell, F. C., & Mello, Z. R. (2007). The reliability and validity of Zimbardo Time Perspective Inventory (ZTPI) scores in academically talented adolescents. Educational and Psychological Measurement, 67, 487–504. doi: 10.1177/0013164406296985 Worrell, F. C., Mello, Z. R., & Buhl, M. (2013). Introducing English and German versions of the Adolescent Time Attitude Scale (ATAS).

Assessment, 4, 496–510. doi: 10.1177/ 1073191110396202 Zhang, J. W., Howell, R. T., & Bowerman, T. (2013). Validating a brief measure of the Zimbardo Time Perspective Inventory. Time and Society, 22, 391–409. doi: 10.1177/0961463X12441174 Zigmond, A. S., & Snaith, R. P. (1983). The Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavica, 67, 361–370. doi: 10.1111/j.1600-0447.1983.tb09716.x Zimbardo, P. G., & Boyd, J. N. (1999). Time perspective: A valid, reliable individual-differences metric. Journal of Personality and Social Psychology, 77, 1271–1288. doi: 10.1037/00223514.77.6.1271 Received November 29, 2014 Revision received June 10, 2015 Accepted July 6, 2015 Published online April 22, 2016 Frank C. Worrell Cognition and Development 4511 Tolman Hall, #1670 University of California Berkeley, CA 94720-1670 USA Tel. +1 (510) 643-4891 E-mail frankc@berkeley.edu

European Journal of Psychological Assessment (2018), 34(1), 41–51

Multistudy Report

A Cross-Cultural Validation of the Implicit Positive and Negative Affect Test (IPANAT) Results From Ten Countries Across Three Continents Markus Quirin,1,2 Monika Wróbel,3 Andrea Norcini Pala,4 Stefan Stieger,5,6 Jos Brosschot,7 Miguel Kazén,8 Joshua A. Hicks,9 Olga Mitina,10 Dong Shanchuan,11 Ruta Lasauskaite,12 Nicolas Silvestrini,12 Patrizia Steca,13 Maria A. Padun,14 and Julius Kuhl8 1

Department of Psychology, Stanford University, USA

2

Department of Psychiatry, Phillips-University Marburg, Germany

3

Institute of Psychology, University of Lodz, Poland

4

Department of Psychiatry, Columbia University, USA

5

Department of Psychology, University of Vienna, Austria Department of Psychology, University of Konstanz, Germany

6 7

Institute of Psychology, Leiden University, The Netherlands

8

Institute of Psychology, University of Osnabrück, Germany

9

Department of Psychology, Texas A&M University, USA

10

Department of Psychology, Lomonosov Moscow State University, Russia

11

School of Psychology, Southwest University, China

12

Department of Psychology, University of Geneva, Switzerland

13

Department of Psychology, University of Milano-Bicocca, Italy Institute of Psychology, Russian Academy of Sciences, Russia

14

Abstract: Self-report measures of affect come with a number of difficulties that can be circumvented by using indirect measurement procedures. The Implicit Positive and Negative Affect Test (IPANAT) is a recently developed measure of automatic activation of representations of affective states and traits that draws on participants’ ratings of the extent to which nonsense words purportedly originating from an artificial language bear positive or negative meaning. Here we compared psychometric properties of this procedure across 10 countries and provide versions in corresponding languages (Chinese, Dutch, English, French, German, Italian, Polish, Russian, and Spanish). The results suggest good reliability, metric invariance, and construct validity across countries and languages. The IPANAT thus turns out as a useful tool for the indirect assessment of affect in different languages and cultures.

Keywords: implicit affect, indirect assessment, positive affect, negative affect, metric invariance

Although self-reports can provide reliable and valid insights into individuals’ inner affective life, they also come with a number of difficulties such as sensitivity to social desirability bias or transparency of research hypotheses (Robinson & Clore, 2002). Therefore, researchers started to develop procedures that measure affect indirectly, such as the Implicit Positive and Negative Affect Test (IPANAT; Quirin, Kazén, & Kuhl, 2009). The IPANAT constitutes a standardized, reliable, brief, and easy-to-apply method for the assessment of implicit affect defined as the automatic activation of cognitive representations of affective experiences European Journal of Psychological Assessment (2018), 34(1), 52–63 DOI: 10.1027/1015-5759/a000315

(Quirin, Kazén, & Kuhl, 2009; see also Leventhal & Scherer, 1987).

Whereas the IPANAT has already been applied in different countries to answer a diversity of research questions (e.g., Hicks & King, 2011; Stieger, Voracek, & Nader, 2014; Yik, Russell, & Steiger, 2011), versions in languages other than German have not yet been validated and compared systematically. Thus, the goal of the present work was to examine psychometric properties of the IPANAT across various languages and cultural contexts and to provide versions of the IPANAT for the use in these languages. Ó 2016 Hogrefe Publishing

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

To do so, research teams from 10 countries applied the IPANAT in their national languages, reported on the factor structure of the IPANAT, and related the IPANAT scores to explicit affect scores. Such an endeavor also bears on the question of universality of affect and will accordingly be discussed here.

After contrasting implicit affect with explicit affect, we provide a detailed description of the IPANAT. Next, we give an overview of the studies conducted in different countries.

Then we present the information on reliability of the IPANAT, replicability of its factor structure, and convergent and discriminant validity across cultures.

Implicit Versus Explicit Measurement of Affect Psychologists have long been interested in measuring affect.

In the 1980s self-report emerged as the most common way to measure individuals’ affective states and traits (Humrichouse, Chmielewski, McDade-Montez, & Watson, 2007). Typically, in the explicit form of assessment, respondents rate the extent to which they have experienced various affective states within a specified time period. Self-report measures of affect are believed to be valid and reliable (for a review, see Humrichouse et al., 2007). However, like other direct measures, they have several limitations which derive from the fact that respondents are openly asked about their feelings. In fact, whereas some individuals can accurately describe their subjective experience, others may either lack the introspective ability to provide correct ratings or alter their responses to conform to social demands (Barrett, Robin, Pietromonaco, & Eyssell, 1998; Robinson & Clore, 2002; Weinberger, Kelner, & McClelland, 1997).

Whereas much research has been directed to developing indirect procedures for the assessment of attitudes (Fazio, Jackson, Dunton, & Williams 1995), self-esteem (Bosson, Swann, & Pennebaker, 2000), motives (Schultheiss & Brunstein, 2001), or stereotypes (Wittenbrink, Judd, & Park, 1997), there is still a considerable need for indirect measures of affect. Such measures could complement direct measures of affect to provide supplementary insight into individuals’ affective processes. Recently, several procedures for the assessment of implicit affect have been introduced (Bartoszek, 2009; Hass, Katz, Rizzo, Bailey, & Moore, 1992; Langens, 2002; Quirin, Kazén, & Kuhl, 2009). Among them, the IPANAT (Quirin, Kazén, & Kuhl, 2009; see also Quirin & Bode, 2014) qualified as a standardized and reliable method that has been validated in a number of diverse settings (e.g., De Visch, 2012; Hicks & King, 2011; Kazén, Kuhl, & Quirin, 2014; Quirin, Kazén, Rohrmann, & Kuhl, 2009; Selcuk, Zayas, Günaydin, Hazan, Ó 2016 Hogrefe Publishing

53

& Kross, 2012; Shimoda, Okubo, Kobayashi, Sato, & Kitamura, 2014; Stieger et al., 2014; Yik et al., 2011).

Following the tradition of research on attitudes, selfesteem, and motives, the output of a procedure aiming at indirectly assessing affect (such as the IPANAT) can be named “implicit affect” (Quirin, Kazén, & Kuhl, 2009; see De Houwer, Teige-Mocigemba, Spruyt, & Moors, 2009, for a differentiation between the terms “indirect” and “implicit”). Accordingly, Quirin, Kazén, and Kuhl (2009) conceptualized implicit affect as the automatic activation of cognitive representations of affective experiences which can but need not become processed consciously. On the basis of the dual-systems model (Strack & Deutsch, 2004), the authors proposed that – analogous to explicit and implicit attitudes – explicit representations of affect are processed in a reflective system, whereas implicit representations of affect are processed in an impulsive system. The reflective system involves deliberative processing based on conceptual classifications, whereas the impulsive system operates on the basis of automatically triggered associations. These two systems may interact at various stages of information processing. Consequently, the reflective system may access information processed in the impulsive system. For instance, implicit affect, if intense enough, can be experienced consciously, which results in self-reported explicit affect. However, reflective, direct access to implicit representations of affect may be limited by cognitive and motivational factors we have already mentioned. Thus, the relationship between implicit and explicit affect, similarly to the relationship between implicit and explicit attitudes (Fazio & Olson, 2003; Hofmann, Gawronski, Gschwendner, Le, & Schmitt, 2005), is weak to moderate (Quirin, Kazén, & Kuhl, 2009).

The Implicit Positive and Negative Affect Test (IPANAT) The IPANAT is an indirect procedure for the assessment of affect, that is, unlike direct measures, it does not rely on people’s ratings of their affective experiences. This procedure is very economical as it can be completed via paperand-pencil in less than 3 min. Participants assess the extent to which six nonsense words (SAFME, VIKES, TUNBA, TALEP, BELNI, & SUKOV), which purportedly originate from an artificial language, express or convey various feelings. To assess their alleged emotional meaning participants use six affectively valenced adjectives. Three adjectives (happy, cheerful, and energetic) refer to positive affect whereas three other (helpless, tense, and inhibited) refer to negative affect, thus forming implicit positive affect and implicit negative affect scales (IPA and INA, respectively; see Quirin, Kazén, & Kuhl, 2009). The two-dimensional European Journal of Psychological Assessment (2018), 34(1), 52–63

54

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

structure of the IPANAT is based on the well-known approach according to which PA and NA are two separate dimensions (Cacioppo & Berntson, 1999; Larsen, McGraw, & Cacioppo, 2001; Watson, Clark, & Tellegen, 1988). The whole test consists of 36 items, with pairs composed of one artificial word and one adjective (6 artificial words 6 adjectives = 36 combinations). The items are scored on a 4-point scale, ranging from doesn’t fit at all to fits very well. In order to conceal the real aim of the IPANAT, the test begins with the instruction which diverts participants’ attention from the fact that their affect is being measured. They are led to believe that it is possible to guess the meaning of artificial words because in all languages there are words that express their meanings by the way they sound (so-called onomatopoeia).

The IPANAT draws on the principle of affect infusion (e.g., Forgas, 1995; Forgas & Bower, 1998) according to which affect exerts an impact on evaluative processes influencing the judgments of unrelated objects. Specifically, in the IPANAT participants’ affect is inferred from their phonological judgments of affectively neutral nonsense words.

Accordingly, implicit affect, once activated, “colors” the phonological judgments of affectively neutral words in the direction of positive or negative affect – participants experiencing higher levels of positive affect make stronger misattribution of positive affective meaning to the artificial words (e.g., associate TUNBA with happy), while those experiencing more negative affect rate the artificial words as conveying negative meaning more strongly (e.g., associate TUNBA with helpless). Thus, the IPANAT scores reflect: (1) state affect (current, short-term fluctuations in affective states, caused by situational factors which influence the judgments of artificial words), (2) trait affect (stable individual differences in the tendency to experience different affective states and, as a result, to ascribe positive/negative meaning to the artificial words), and (3) subjective associations that artificial words may evoke (error variance).

Previous research has shown that, although participants deliberately rate the fit between an artificial word and a positive or negative adjective (as they would in a typical self-report), more than 98% of them believe in the cover story and do not suspect that the test measures affect (Quirin, Kazén, & Kuhl, 2009). Following the reflectiveimpulsive model by Strack and Deutsch (2004), we assume that in responding to the IPANAT both the reflective and impulsive systems are simultaneously active – the reflective system in carrying out the intended phonological judgments 1

and the impulsive system in creating associative bonds between an artificial word and a mood adjective. As such, the IPANAT measures affect implicitly not in terms of involuntary behavioral response but in terms of the uncontrolled influence of affective processes on this response. In this regard, the IPANAT is similar to the affect misattribution procedure that implicitly measures attitudes toward objects on the basis of how participants rate Chinese ideographs primed by these objects (Payne, Cheng, Govorun, & Stewart, 2005). This qualifies the IPANAT as an indirect measure of affect.

Psychometric properties of the IPANAT have been tested in a series of studies (Quirin, Kazén, & Kuhl, 2009). The analyses yielded two separate factors that showed high internal consistency, test-retest reliability, and construct validity. Cronbach’s alphas for both IPA and INA were .81, which is high for an implicit test. Correlation coefficients between the scores across 1-week, 1-month, 2-month, and 12-month intervals ranged from .60 to .76, indicating that there is a strong trait component in both IPA and INA. The IPANAT scores correlated positively but moderately with explicit measures of affect, for example, broadly used Positive and Negative Affect Schedule (PANAS; Watson et al., 1988), as well as other measures of positive and negative affectivity: whereas IPA was positively associated with explicit positive affect and extraversion, INA was positively associated with explicit negative affect and neuroticism. Both IPA and INA, similarly to the PANAS (Watson, 2000; Watson & Clark, 1994), were unrelated to participants’ gender.1 Moreover, the IPANAT turned out to be sensitive to changes in affect after the presentation of affective stimuli, which confirmed that the test can also be used for measuring state affect.

Subsequent studies provided further evidence for the reliability of the test and the separability of its dimensions. For example, whereas IPA but not INA predicted circadian variations in cortisol, INA but not IPA predicted cortisol responses to experimentally induced stress (Quirin, Kazén, Rohrmann, et al., 2009). In a study on the interaction between implicit affect and personality, INA interacted with impaired emotion regulation abilities (“state orientation”) to predict a tendency toward analytic processing, whereas IPA interacted with high emotion regulation abilities (“action orientation”) to predict holistic processing (Kazén et al., 2014). Importantly, in each of the above-mentioned studies measures of explicit affect did not significantly predict performance.

Some research suggests that men and women differ on positive and negative affect scales (e.g., Crawford & Henry, 2004). Although these gender differences in affect seem compatible with common beliefs (Fabes & Martin, 1991; Shields, 2002) and with gender differences reported in personality traits (e.g., Costa, Terracciano, & McCrae, 2001), they did not emerge in US normative samples. For instance, Watson and Clark (1994) found no differences between men and women on the PANAS-X scales in 10 large samples including over 8,000 participants (see also Watson, 2000). Similar results were found for the earlier version of the scale (the PANAS; Watson et al., 1988) and for positive and negative emotions assessed with structured diaries (Oatley & Duncan, 1994).

European Journal of Psychological Assessment (2018), 34(1), 52–63

Ó 2016 Hogrefe Publishing

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

Overview of the Present Research

55

Table 1. Demographic characteristics of the samples Gender

Although most of the studies using the IPANAT in various settings have been conducted in different countries, the test has not been validated cross-culturally yet. Hence, the comparability of the data collected with the IPANAT in various cultural contexts has not been evaluated. Over the last decades, with a growing awareness of cross-cultural issues, multi-country validation studies have become more and more popular. The reason for their growing popularity is that they enable psychologists to compare research findings from different countries and different languages (Church et al., 2011; Ziegler & Bensch, 2013). For instance, in the explicit affect domain, cross-cultural explorations have shed some light on the universality of a two-dimensional structure of affect. Although over the years the orthogonality of positive and negative affect has been a controversial issue (Crawford & Henry, 2004; Feldman Barrett & Russell, 1998; Schmukle, Egloff & Burns, 2002) and in some populations (e.g., French-Canadian, African American) alternative models were found (Gaudreau, Sanchez, & Blondin, 2006; Merz, et al., 2013), cross-cultural studies have demonstrated that positive and negative affect dimensions are highly generalizable to other languages and cultures (Almagor & Ben-Porath, 1989; Ayuso-Mateos et al., 2013; Thompson, 2007). Furthermore, other characteristics of explicit affect (e.g., correlations with personality factors such as extraversion and neuroticism or manifestations through both short-term transient feelings and relatively stable individual traits) have also been replicated across diverse cultural settings (e.g., Brzozowski, 2010; Gyollai, Simor, Köteles, & Demetrovics, 2011; Krohne, Egloff, Kohlmann, & Tausch, 1996; Melvin & Molloy, 2000; Robles & Páez, 2003; Terracciano, McCrae, & Costa, 2003).

Similar evidence concerning the universality of implicit affect is missing. One might expect that due to the fact that both the impulsive and reflective systems access the same affective experience, implicit affect and explicit affect should share the same universal characteristics. Yet, it remains unknown whether the IPANAT scores would reflect this universality. Therefore, we conducted the present study to assess the internal consistency, construct validity, and cross-cultural factorial invariance of the IPANAT across various countries. To that aim, we coordinated a common study design with five teams from different countries (Austria, Poland, Russia, the USA, and Uzbekistan). Another five teams (from China, Italy, Mexico, the Netherlands, and Switzerland) had previously collected some data on the IPANAT. Because these studies provide translations of the scale and the data are informative we additionally included these data in our analyses, although the designs of these studies somewhat differed from the coordinated design. Ó 2016 Hogrefe Publishing

n Austria 115 China 289 Italy 955 Mexico 119 The Netherlands 1,123 Poland 288 Russia 513 Switzerland 57 USA 128 Uzbekistan 168 Total sample 3,755

Age (years)

% Women % Men % Missing 62.6 69.6 51.0 85.7 85.6 80.2 76.8 82.5 70.3 57.7 71.4

37.4 28.4 48.5 14.3 10.8 19.8 21.2 17.5 29.7 41.1 26.9

– 2.0 0.5 – 3.6 – 1.9 – – 1.2 1.7

M

SD

30.69 11.42 20.49 1.72 29.51 6.71 21.78 5.50 46.29 10.92 22.13 3.53 21.15 6.21 21.95 5.01 18.91 1.06 22.49 5.77 30.95 12.98

Based on the previous findings by Quirin, Kazén, and Kuhl (2009), we predicted that IPA and INA would represent two separate dimensions and that participants would score higher on IPA than on INA. Moreover, we expected that the IPANAT scores would be relatively stable over time, thereby, besides a state component, they would reflect a trait component of implicit affect as well. We also hypothesized that the structure of the IPANAT would be cross-culturally invariant and that IPA would be positively related to explicit positive affect, whereas INA would be positively related to explicit negative affect.

Materials and Method Participants The total sample (N = 3,755) consisted of participants from Austria, China, Italy, the Netherlands, Mexico, Poland, Russia, Switzerland, the USA, and Uzbekistan. Of these participants, 2,682 (71.4%) were women, 1,009 (26.9%) were men, and 64 (1.7%) did not report their gender. The number of participants, percentage of men and women, and mean age for each country are reported in Table 1. Both the proportion of men and women, w2(9) = 414.02, p < .001, V = .33, and mean age differed significantly, F(9, 3,688) = 746.26, p < .001, η2 = .65, across countries.

Procedure The IPANAT instruction and mood adjectives were translated from English or German into the local languages (see Table 2). In all samples the translation procedures were carried out by native speakers with a fluent command of German or English. In the Netherlands, Mexico, Poland, Russia, and Uzbekistan the translations were additionally evaluated by a group of at least three judges who rated the clarity of the instructions and the understandability of European Journal of Psychological Assessment (2018), 34(1), 52–63

56

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

Table 2. Mood adjectives used in different languages Original mood adjectives Austria

Happy

Cheerful

Energetic

Helpless

Tense

Inhibited

gut gelaunt

fröhlich

aktiv

hilflos

verkrampft

gehemmt

felice feliz goedgehumeurd zadowolony

allegro alegre vrolijk radosny

energico activo actief pełen energii

impotente desamparado hulpeloos bezradny

nervoso tenso gespannen spięty

inibito inhibido geremd zahamowany

heureux happy

joyeux cheerful

plein d’énergie energetic

désespéré helpless

tendu tense

inhibé inhibited

China Italy Mexico The Netherlands Poland Russia/Uzbekistan Switzerland USA

Note. The full version of the IPANAT can be found in Quirin, Kazén, and Kuhl (2009).

the adjectives. Moreover, the teams from the Netherlands and Poland developed new sets of artificial words following the procedure by Quirin, Kazén, and Kuhl (2009) because the original words turned out to bear too much affective meaning for Polish and Dutch language users.2 Participants were recruited via e-mailing lists, leaflets, social networking services, and word-of-the-mouth advertising. All data were collected either via paper-and-pencil or web-based surveys. The former procedure was applied in Austria, China, Mexico, the Netherlands, Russia, Uzbekistan, and the USA, whereas the latter in Italy and Switzerland. In Poland both methods were used. The majority of participants (n = 211) were administered paper-and-pencil measures, whereas the other (n = 77) were tested via the Internet. In all samples the IPANAT was administered as the first measure in order not to raise expectations that the test measures affect. Having completed the IPANAT, participants assessed their explicit affect and answered demographic questions. In three countries we also collected the data on time stability of the IPANAT. In Austria (n = 115) the test was administered two more times (after one-week and after four-week intervals), whereas in Poland (n = 119) and Mexico (n = 101) one more time (after 4 weeks and after 30 min, respectively).

whereas in Mexico and Switzerland participants rated their momentary affective state. The Dutch team did not collect data on explicit affect. In the majority of countries we used the local versions of the Positive and Negative Affect Schedule (PANAS; Watson et al., 1988)3 which is a widely applied 20-item self-report measure of affective states and traits. The items were scored on a 5-point scale from 1 (= very slightly or not at all) to 5 (= extremely). In the Mexican sample, EPA and ENA were measured with the same mood adjectives as included in the IPANAT. Participants were asked to report the extent to which they felt happy, cheerful, energetic, helpless, tense, and inhibited on a rating scale from 1 (= not at all) to 4 (= completely).

By analogy to the IPANAT, we composed EPA and ENA scales computing average scores for positive adjectives and negative adjectives, respectively. In Poland both the PANAS and mood adjectives were used (in two independent samples). Finally, in Switzerland the short form of the hedonic tone scale of the UWIST Mood Adjective Checklist (Matthews, Jones, & Chamberlain, 1990) translated into French was applied. The scale has been reported to correlate positively with energetic arousal and negatively with tense arousal (Matthews et al., 1990). Participants reported how they felt right now on a rating scale from 1 (= not at all) to 7 (= very much).

Measures Explicit positive affect (EPA) and explicit negative affect (ENA) were assessed with three different yet related instruments. In the samples from the five start-up studies (Austria, Poland, Russia, Uzbekistan, and the USA) we asked participants to indicate how they felt in general. Participants of the Chinese and Italian samples rated their affective states within last week or last month, respectively, 2

3

Results Descriptive Statistics, Intercorrelations, Internal Consistency, and Time Stability Descriptive statistics for IPA and INA are reported in Table 3. The kurtosis and skewness values indicated that

Polish artificial words were: KOFGE, BASDI, MIKUF, LOPEW, AHYKO, and CEMJU. Dutch artificial words were: TOMAK, RAKEG, NABEN, HASWI, POGIS, and TEKOD. Chinese version: Qiu, Zheng, and Wang (2008); Italian version: Terracciano et al. (2003); German version: Krohne, Egloff, Kohlmann, and Tausch (1996); Polish version: Brzozowski (2010); Russian version: Osin (2012).

European Journal of Psychological Assessment (2018), 34(1), 52–63

Ó 2016 Hogrefe Publishing

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

57

Table 3. Mean scores and Cronbach’s α of IPA and INA Differences between mean IPA and INA Countries

Scales

Austria

IPA INA IPA INA IPA INA IPA INA IPA INA IPA INA IPA INA IPA INA IPA INA IPA INA

China Italy Mexico The Netherlands Poland Russia Switzerland USA Uzbekistan

IPA and INA correlation

Cronbach’s α

M

SD

Min

Max

Skewness

Kurtosis

t

df

p

Cohen’s d

.78 .86 .79 .85 .88 .81 .81 .78 .89 .86 .83 .76 .83 .69 .72 .65 .85 .82 .75 .67

2.34 1.94 2.10 1.97 1.86 1.73 2.27 1.87 2.08 2.00 2.32 2.21 2.25 2.04 2.23 1.96 2.03 1.98 2.18 2.02

0.42 0.45 0.44 0.45 0.49 0.44 0.47 0.43 0.49 0.45 0.40 0.39 0.48 0.42 0.38 0.35 0.37 0.36 0.47 0.41

1.22 1.00 1.11 1.00 1.00 1.00 1.22 1.00 1.00 1.00 1.17 1.22 1.00 1.00 1.50 1.33 1.28 1.11 1.00 1.00

3.83 3.56 3.83 3.83 4.00 4.00 3.61 3.17 4.00 4.00 3.61 3.56 3.73 3.07 3.00 2.83 3.17 2.83 3.47 3.13

.40 .22 .02 .20 .78 .66 .40 .51 .76 .59 .06 .04 .05 .05 .01 .21 .40 .09 .08 .12

1.48 0.51 0.21 0.21 1.27 0.77 0.18 0.11 1.56 1.21 0.46 0.45 0.01 0.40 0.70 0.53 0.05 0.53 0.14 0.18

6.87

114

.001

1.29

.03 (ns)

.08 (ns)

4.19

288

.001

0.49

.25***

.01 (ns)

9.19

940

.001

0.60

.57***

.18***

7.40

118

.001

1.36

.14 (ns)

.10 (ns)

4.29 1097

.001

0.26

.12***

.02 (ns)

3.46

287

.001

0.41

.02 (ns)

.02 (ns)

7.55

510

.001

0.67

.05 (ns)

.05 (ns)

4.21

56

.001

1.13

.08 (ns)

.05 (ns)

1.29

127

.198

0.23

.25**

.13 (ns)

3.78

164

.001

0.59

.28***

.17*

r

rpartial

Notes. Partial correlation coefficients were computed using energetic and tense as controls. *p < .05; **p < .01; ***p < .001.

the scores tended to be normally distributed in the majority of samples. In all but one sample, participants reported significantly higher mean levels of IPA than INA, which suggested that on average they tended to judge artificial words as conveying more positive than negative meaning.

The only exception was the American sample where no significant differences between mean IPA and INA scores were observed (although numerically IPA scores were higher than INA scores). Consistent with our predictions, in the Austrian, Mexican, Polish, Russian, and Swiss samples, IPA and INA were unrelated. In China, Italy, the Netherlands, the USA, and Uzbekistan, however, we observed unexpected positive correlations between IPA and INA. Nevertheless, after controlling for two arousalrelated adjectives, namely energetic and tense, these IPA-INA correlations dropped to nonsignificance (in the Chinese, Dutch, and American samples) or became significantly weaker (in the Italian sample; Z = 10.16; p < .001).

The only exception was the Uzbek sample where no significant decrease of the correlation coefficient was observed, Z = 1.05; p = .292.

Gender differences were significant neither for IPA nor for INA (ts < 1.80; ps > .10) with the exception of the Italian sample, where women scored higher on the IPA scale than men t(934) = 2.07; p = .039; and the Swiss sample where women scored lower on the INA scale than men, t(55) = 2.17; p = .034. Yet, these differences represented small- (Cohen’s d = .14) and medium-sized effects (d = .58), respectively.

Ó 2016 Hogrefe Publishing

The internal consistencies, using Cronbach’s alphas, ranged across countries from .72 to .89 for IPA and from .67 to .86 for INA (see Table 3). In Austria, China, Italy, Mexico, the Netherlands, Poland, and the USA, the coefficients were similar to those found in the German sample (Quirin, Kazén, & Kuhl, 2009), whereas in Russia, Switzerland, and Uzbekistan, they were somewhat lower but still acceptable.

Additionally, on the basis of the data collected in the Austrian (n = 115), Mexican (n = 119), and Polish samples (n = 101), we computed test-retest reliabilities. In Austria the correlations between the scores were .57 for IPA and .65 for INA (across a one-week interval) and .55 for IPA and .60 for INA (across a four-week interval). In Poland the test-retest correlations were .68 for IPA and .67 for INA (across four weeks), whereas in Mexico they were .74 for IPA and .69 for INA (across 30 min). Overall, these results confirm that the IPANAT scores are psychometrically solid and predominantly reflect a stable, trait component of positive and negative affect.

Cross-Cultural Measurement Invariance To investigate the measurement equivalence (i.e., the equality of factor loadings, item intercepts, and residual variances) of the IPANAT across diverse cultures, we performed a multiple group confirmatory factor analysis (MG-CFA; Mplus 6.1; Muthén & Muthén, 2010). Specifically, we analyzed (a) weak (metric) invariance (the equality

European Journal of Psychological Assessment (2018), 34(1), 52–63

58

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

Table 4. CFA of bi-factorial model of the IPANAT by country Countries

CFI

TLI

RMSEA

w

df

p

Austria China Italy Mexico The Netherlands Poland Russia Switzerland The USA Uzbekistan

1 0.99 0.99 1 0.98 1 0.98 0.69 0.99 0.95

1 0.98 0.97 1 0.96 1 0.97 0.42 0.97 0.91

< .05 .03 .05 < .05 .07 < .05 .06 .22 .06 .11

6.40 10.39 29.79 7.51 55.48 7.35 20.65 30.94 11.17 22.95

8 8 8 8 8 8 8 8 8 8

ns ns .001 ns .001 ns .001 .001 ns .001

2

< < < < <

w /df ratio 2

0.80 1.30 3.72 0.94 6.93 0.92 2.58 3.87 1.39 2.87

of factor loadings across groups), (b) strong (scalar) invariance (the equality of item intercepts across groups), and (c) – if metric and scalar invariance was achieved – strict invariance (the equality of item residuals across groups; Byrne, Shavelson, & Muthén, 1989).

The invariance across groups was tested according to Muthén and Muthén (2009, 2010). First, a confirmatory factor analysis (CFA) was conducted on the overall sample and on each country individually. Then, three nested models were compared: model 1 with free loadings, intercepts, and residual variances (i.e., none of the three parameters were considered invariant); model 2, where only the loadings were constrained and held equal across countries (metric invariance); and model 3, with loadings and intercepts held equal across countries (metric and scalar invariance).

The w2 of the nested models (model 1 vs. model 2, and model 2 vs. model 3) were compared using the DIFFTEST option. We used the maximum likelihood parameter estimates with standard errors and a mean- and varianceadjusted w2 test statistic that are robust to non-normality (MLMV; Muthén & Muthén, 2010).

The CFA as ran on the overall sample confirmed the goodness of fit of the model (CFI = .99; TLI = .98, RMSEA = .046, w2/df = 8.77), nonetheless the w2/df ratio was over the cutoff of 3.0. Hence, we decided to test the model fit within each country separately (see Table 4).

The results presented in Table 4 show that the bi-factorial model was confirmed in all countries except for Switzerland. Therefore, we tested invariance after excluding the Swiss sample (see Table 5). The DIFFTEST suggests that constraining the factor loadings in model 2 and the loadings and intercepts in model 3 worsened the model fit significantly (as compared to model 1 in which the parameters were unconstrained). However, CFI, TLI, and RMSEA of model 2 slightly improved. Constraining the intercepts (model 3) resulted in poor goodness-of-fit indices.

Because the model was not fully invariant across the analyzed countries we proceeded with the test of partial invariance, in which a subset of the parameters was not constrained to be invariant (Muthén & Muthén, 2009).

European Journal of Psychological Assessment (2018), 34(1), 52–63

Table 5. Goodness-of-fit indices – test of invariance of three nested models DIFF TEST

Model 1 Model 2 Model 3

CFI

TLI

RMSEA

w2

df

p

.98 .98 .92

0.97 0.98 0.91

.06 .05 .10

– 51.06 573.824

– 32 32

– .018 < .001

Note. Model 1 – free loadings, free intercepts, and free uniqueness; Model 2 – free intercepts and free uniqueness; Model 3 – free uniqueness.

The modification indices in model 2 suggest that residuals’ covariance between energetic and tense in the Dutch sample may improve the fit, which was confirmed in a further analysis (CFI = .98, TLI = .98, RMSEA = .048, DIFFTEST: w2 = 21.06, df = 31, p = .91; cf. the fit of nested models 1 and 2 in Table 5). Thus, model 3 was run including the covariance as in model 2. Modification indices > 50 suggest that freeing the intercepts of helpless in China, Poland, and Uzbekistan, as well as helpless and tense in Russia would improve the fit (all the other intercepts were still held equal across countries). Although the indices improved significantly, the DIFFTEST was still statistically significant (CFI = .97, TLI = .96, RMSEA = .064, DIFFTEST: w2 = 192.52, df = 27, p < .001; cf. the fit of nested models 2 and 3 in Table 5), which suggests that the item intercepts cannot be considered invariant. Thus, because scalar invariance was not confirmed we did not test strict invariance, that is the equality of variance residuals (Byrne et al., 1989).

In sum, following Muthén and Muthén (2009), who suggested that invariance of intercepts is quite difficult to achieve, we conclude that the IPANAT is metrically invariant across countries (except for Switzerland), albeit we failed to achieve scalar invariance. In other words, factor loadings are the same across countries (in a way that relationships between IPA, INA, and other constructs can be examined cross-culturally), but the countries cannot be compared in terms of the IPANAT mean scores.

Convergent and Discriminant Validity Table 6 presents Pearson’s correlations of IPA and INA with explicit affect. As expected according to previous findings (Quirin, Kazén, & Kuhl, 2009), in the majority of samples IPA was positively related to EPA and unrelated to ENA, whereas INA was positively related to ENA and unrelated to EPA. The only exceptions were the Austrian and Uzbek samples, in which some links were only marginally significant, and the Italian and Chinese samples, in which positive correlations between INA and EPA were observed.

These correlations, however, turned out to be nonsignificant after controlling for tense and energetic (in the Chinese sample: rpartial (285) = .06; p = .328; in the Italian Ó 2016 Hogrefe Publishing

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

59

Table 6. Pearson’s correlations between implicit and explicit affect Explicit affect

Countries

n

Implicit affect IPA r

EPA (PANAS)

Austria China Italy Poland Russia USA Uzbekistan EPA (same adjectives) Mexico Poland ENA (PANAS) Austria China Italy Poland Russia USA Uzbekistan ENA (same adjectives) Mexico Poland Hedonic tone (UMACL) Switzerland

115 289 939 77 511 128 165 114 211 115 289 939 77 511 128 165 114 211 57

.16 .21 .17 .38 .17 .27 .14 .33 .14 .01 .06 .01 .03 .03 .02 .08 .11 .08 .30

INA p

.084 <.001 <.001 <.001 <.001 .002 .068 <.001 .037 .887 .309 .887 .775 .524 .824 .301 .263 .259 .024

r

p

.08 .374 .15 .012 .08 .009 .05 .640 .02 .790 .14 .110 .03 .752 .11 .230 .11 .125 .18 .054 .28 <.001 .16 <.001 .21 .066 .22 <.001 .22 .012 .21 .008 .32 .001 .14 .049 .24 .072

Notes. The Netherlands is not listed because we did not measure explicit affect in this sample.

sample: rpartial (931) = .04; p = .201). Additionally, in the Swiss sample, hedonic tone was positively linked to both IPA and INA. Yet, the correlation with INA reached only marginal significance and dropped to non-significance when tense was controlled for, rpartial (57) = .07, p = .598.

Discussion The aim of the present research was to assess psychometric properties of the IPANAT across various cultural contexts.

The results of the studies conducted in Austria, China, Italy, Mexico, the Netherlands, Poland, Russia, Switzerland, the United States, and Uzbekistan confirm that the IPANAT is able to measure implicit affect in various countries and languages.

The IPANAT scales have good internal consistency (for IPA and INA about .81 and .78, respectively) that is comparable to or even higher than the reliability of other commonly used implicit measures (Fazio & Olson, 2003; Krause, Back, Egloff, & Schmukle, 2011) and only somewhat lower than the reliability of self-report mood scales (e.g., PANAS or UMACL). The IPANAT scores are also quite stable over time, which confirms that the test is suitable for assessing positive and negative affect treated as relatively stable personality characteristics (positive and negative affectivity; Watson, 2000). At the same time, moderate test-retest correlations suggest that the IPANAT scores reflect not only trait but also state variance. Ó 2016 Hogrefe Publishing

In line with previous studies (Quirin, Kazén, & Kuhl, 2009; De Visch, 2012), the present research also revealed that people from different countries tend to attribute more positive than negative meaning to the artificial words used in the IPANAT. This result not only reflects a so-called “positivity offset” (Cacioppo & Berntson, 1994) but also corresponds with the observation that most of the time people experience pleasant rather than unpleasant affective states (Watson, 2000). The current research extends the validity of this observation beyond explicit affect. Moreover, the IPANAT scales are either unrelated or weakly related to participants’ gender, which is consistent with the studies showing that men and women report similar levels of explicit positive and negative affect (Watson, 2000; Watson & Clark, 1994).

Although in some of the samples IPA and INA were unrelated, in other countries an unexpected positive correlation between these two dimensions was observed. As this relationship appears to differ across cultures, we cannot exclude that it is affected by cultural factors. For instance, one of the reasons why independence between two scales may shift toward positive correlation is acquiescence (Paulhus & Vazire, 2007). It has been found that this response bias exerts moderate influence on explicit affect ratings (Diener, Smith, & Fujita, 1995; Watson, 2000).

Moreover, it differs across countries. Specifically, it has been observed more often in collectivist, large-powerdistance cultures than individualist, small-power-distance ones (Johnson, Kulesa, Cho, & Shavitt, 2005; Van Herk, Poortinga, & Verhallen, 2004). In our research, however, only two samples in which IPA and INA were positively correlated are marked by collectivism and large power distance (China, Uzbekistan). The rest of the samples in which we observed a positive IPA-INA correlation (Italy, the Netherlands, the USA) are characterized by high individualism and small power distance (Hofstede, Hofstede, & Minkov, 2010). Therefore, the differences in relationship between IPA and INA across the samples probably arise from other reasons.

One of these reasons may be that various cultures attribute slightly different meaning to mood adjectives, as it has already been found for some adjectives referring to personality (Nye, Roberts, Saucier, & Zhou, 2008). When subjected to confirmatory factor analysis (CFA), the IPANAT failed to show scalar invariance, which implies that individuals in one sample may score differently on some items than individuals in another sample because of the way they conceptualize the meaning of these items and not because of true mean differences. In addition, the analysis of partial correlations showed that two adjectives, namely energetic and tense, could be responsible for the IPA-INA correlation in some samples. A closer examination of the connotations of these two adjectives suggests that they may share a European Journal of Psychological Assessment (2018), 34(1), 52–63

60

common semantic component as they are both associated with activation or arousal (Thayer, 1996, 2001; Yik et al., 2011). Thus, it is possible that in some countries these two items may overlap although they originate from separate subscales. Even though currently it is difficult to explain what specific cultural factors are responsible for this overlap, this finding points out to the importance of subtle differences in the conceptualizations of positivity and negativity that may appear across cultures. For instance, when validating the short form of the PANAS in the Australian sample, Mackinnon et al. (1999) found that the item excited significantly correlated with both EPA and ENA. Moreover, the pleasure-displeasure dimension of Russell’s (1980) circumplex model has been replicated across cultures, whereas its arousal dimension has not consistently emerged in data (Russell, Lewicka, & Niit, 1989). This suggests that in some cultures certain mood adjectives – especially those associated with activation or arousal – may carry ambiguous meaning (see also Thompson, 2007). It seems likely that the IPANAT may be more sensitive to weak differences in meaning of mood adjectives than explicit measures.

The correlations of the IPANAT scores with the PANAS and other explicit affect scales support this conclusion.

The examples of the Chinese, Italian, and Swiss samples suggest that tense and energetic should be controlled for when analyzing the relationship between implicit and explicit positive affect. Consequently, the possibility of comparing the means of the IPANAT across various countries is reduced as the discrepancies observed across various samples may in fact result from different conceptualizations of mood adjectives that refer to activation or arousal. Thus, it remains an open question whether (and if so, to what extent) the level of implicit positive and negative affect differs across countries. Nevertheless, the test showed good fit tested both in the overall sample encompassing many different countries (and languages) and in each sample separately. Moreover, in the multiple-group analysis, the metric invariance was achieved, which implies that the two factors (IPA and INA) can be tested in relation to other constructs cross-nationally.

In two samples (Italian and Swiss) we did not find support for the separability of the IPANAT subscales. In the Italian sample the positive correlation between IPA and INA was higher than in other samples (where it was consistently nonsignificant or low), whereas in the Swiss sample the two-dimensional model of affect was not confirmed in the CFA. Importantly, in these two samples but not in the other samples the IPANAT was filled out exclusively online, which suggests that Internet-based administration might have influenced the results. This explanation is in line with the fact that we found gender differences in the IPANAT scores only in these two samples (although such differences did not appear in the rest of the countries). It is known that European Journal of Psychological Assessment (2018), 34(1), 52–63

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

gender can affect web-based surveys to a greater extent than paper-and-pencil ones (Kay, Gathercoal, & Buhrow, 2011). Additionally, it cannot be excluded that Internetbased administration of the IPANAT results in so-called “careless responding” (Johnson, 2005; Ward & Pond, 2013), which refers to the situation in which participants respond to a survey in a manner that does not reflect their true scores. Careless responding may occur when the researcher and participant are physically disconnected, which is inherent to web-based survey research (Ward, 2015). At the moment, however, it is impossible to say whether the IPANAT scores were influenced by the webbased research format (e.g., in the part of the Polish sample, where we also used an Internet-based survey, we found neither positive IPA-INA correlation nor gender differences).

Nevertheless, this issue should be investigated more thoroughly in the future.

Finally, some limitations of the present research should be mentioned. We cannot rule out that differences in procedures applied in the countries involved in the project contributed to some of the observed discrepancies. First, as we have already mentioned the data in all 10 samples were collected either via paper-and-pencil or the Internet, which might have influenced the results. Thus future research would benefit from comparing these two alternative survey formats of the IPANAT more systematically. Second, there were demographic dissimilarities across samples and women were overrepresented in almost all of them.

Although in eight out of 10 samples we found no gender effects, the examples of the Italian and Swiss samples show that the relation between gender and implicit affect should be analyzed more carefully in the future. Third, explicit measures of affect we used differed across samples in terms of the time frames involved (starting from momentary affect and ending with relatively stable affectivity). This could have influenced the relationship between implicit and explicit affect as the IPANAT has a strong trait component (Quirin, Kazén, & Kuhl, 2009, Quirin, Kazén, Rohrmann, et al., 2009). Future studies would thus benefit from analyzing the relationship between implicit and explicit state affect as well as implicit and explicit trait affect more thoroughly.

Overall, the present research supports the reliability and construct validity of the IPANAT demonstrating that implicit affect can be successfully measured in various cultural contexts. The IPANAT fills the need for a measure that assesses affect in an indirect instead of direct way. Therefore, it enables researchers based in different countries to investigate the discrepancies between explicit and implicit affect and their correlates. Measuring affect on a variety of levels (explicit, implicit, and psychophysiological) provides a better understanding of a range of psychological processes, as it has been already demonstrated for cortisol Ó 2016 Hogrefe Publishing

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

regulation (Quirin, Kazén, Rohrmann, et al., 2009), repeated subliminal exposure (Hicks & King, 2011), or analytic versus holistic processing (Kazén et al., 2014).

Moreover, the ease of use of the IPANAT opens up possibilities for applying similar procedures to other types of emotional constructs, such as implicit motives, or implicit prejudice (Quirin, Kuhl, Luckey, Pyszczynski, & Bode, 2014).

Acknowledgments Markus Quirin and Monika Wróbel share first authorship for this article. This research was supported by grants from the Deutsche Forschungsgemeinschaft (QU 274 4-1) awarded to M. Quirin, the National Science Centre in Poland (DEC-2012/07/D/HS6/00947) awarded to M. Wróbel, the Swiss National Science Foundation (SNF 100014-122604/1 and 100014-131760/1) awarded to G. H. E. Gendolla, the Netherlands Organization for Health Research and Development (ZON-MW, 40-0081 2-98-1 I 029) awarded to J. F. Brosschot, and the Russian Foundation for Basic Research (N. 14-06-00316) awarded to A. Tkhostov.

References Almagor, M., & Ben-Porath, Y. S. (1989). The 2-factor model of self-reported mood: A cross-cultural replication. Journal of Personality Assessment, 53, 10–21.

Ayuso-Mateos, J. L., Miret, M., Caballero, F. F., Olaya, B., Haro, J. M., Kowal, P., & Chatterji, S. (2013). Multi-country evaluation of affective experience: Validation of an abbreviated version of the Day Reconstruction Method in seven countries. PLoS One, 8, e61534.

Barrett, L. F., Robin, L., Pietromonaco, P. R., & Eyssell, K. M. (1998). Are women the “more emotional” sex? Evidence from emotional experiences in social context. Cognition and Emotion, 12, 555–578.

Bartoszek, G. (2009). Implicit measure of emotions: Distinguishing among emotions of the same valence, Chicago, IL: University of Illinois. Retrieved from http://indigo.uic.edu/bitstream/handle/ 10027/9163/Bartoszek_Greg-ory.pdf?sequence=1 Bosson, J. K., Swann, W., & Pennebaker, J. W. (2000). Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? Journal of Personality and Social Psychology, 79, 631–643.

Brzozowski, P. (2010). Skala uczuć pozytywnych i negatywnych (SUPIN). Polska adaptacja PANAS [Positive and Negative Affect Scales: The Polish adaptation]. Warsaw, Poland: Pracownia Testów Psychologicznych.

Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456–466.

Cacioppo, J. T., & Berntson, G. G. (1994). Relationship between attitudes and evaluative space: A critical review, with emphasis on the separability of positive and negative substrates.

Psychological Bulletin, 115, 401–423.

Ó 2016 Hogrefe Publishing

61

Cacioppo, J. T., & Berntson, G. G. (1999). The affect system: Architecture and operating characteristics. Current Directions in Psychological Science, 8, 133–137.

Church, A. T., Alvarez, J. M., Mai, N. T. Q., French, B. F., Katigbak, M. S., & Ortiz, F. A. (2011). Are cross-cultural comparisons of personality profiles meaningful? Differential item and facet functioning in the Revised NEO Personality Inventory. Journal of Personality and Social Psychology, 101, 1068–1089.

Costa, P. T. Jr., Terracciano, A., & McCrae, R. R. (2001). Gender differences in personality traits across cultures: Robust and surprising findings. Journal of Personality and Social Psychology, 81, 322–331.

Crawford, J. R., & Henry, J. D. (2004). The positive and negative affect schedule (PANAS): Construct validity, measurement properties and normative data in a large non-clinical sample.

British Journal of Clinical Psychology, 43, 245–265.

De Houwer, J., Teige-Mocigemba, S., Spruyt, A., & Moors, A. (2009). Implicit measures: A normative analysis and review.

Psychological Bulletin, 135, 347–368.

De Visch, K. (2012). The significance of hidden feelings, Validation of the IPANAT in two samples and its relationship with creativity.

Ghent, Belgium: Ghent University. Retrieved from http://lib.ugent. be/fulltxt/RUG01/001/893/964/RUG01- 001893964_2012_0001_ AC.pdf Diener, E., Smith, H., & Fujita, F. (1995). The personality structure of affect. Journal of Personality and Social Psychology, 69, 130–141.

Fabes, R. A., & Martin, C. L. (1991). Gender and age stereotypes of emotionality. Personality and Social Psychology Bulletin, 17, 532–540.

Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J. (1995).

Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69, 1013–1027.

Fazio, R. H., & Olson, M. A. (2003). Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297–327.

Feldman Barrett, L., & Russell, J. A. (1998). Independence and bipolarity in the structure of current affect. Journal of Personality and Social Psychology, 74, 967–984.

Forgas, J. P. (1995). Mood and judgment: The affect infusion model (AIM). Psychological Bulletin, 117, 39–66.

Forgas, J. P., & Bower, G. H. (1998). Affect in social judgments.

Australian Journal of Psychology, 81, 1001–1013.

Gaudreau, P., Sanchez, X., & Blondin, J. (2006). Positive and negative affective states in a performance-related setting: Testing the factorial structure of the PANAS across two samples of French-Canadian participants. European Journal of Psychological Assessment, 22, 240–249.

Gyollai, A., Simor, P., Köteles, F., & Demetrovics, Z. (2011).

Psychometric properties of the Hungarian version of the original and the short form of the Positive and Negative Affect Schedule (PANAS). Neuropsychopharmacologia Hungarica, 13, 73–79.

Hass, R. G., Katz, I., Rizzo, N., Bailey, J., & Moore, L. (1992). When racial ambivalence evokes negative affect, using a disguised measure of mood. Personality and Social Psychology Bulletin, 18, 786–797.

Hicks, J. A., & King, L. A. (2011). Subliminal mere exposure and explicit and implicit positive affective responses. Cognition and Emotion, 25, 726–729.

Hofmann, W., Gawronski, B., Gschwendner, T., Le, H., & Schmitt, M. (2005). A meta-analysis on the correlation between the Implicit Association Test and explicit self-report measures. Personality and Social Psychology Bulletin, 31, 1369–1385.

Hofstede, G., Hofstede, G. J., & Minkov, M. (2010). Cultures and Organizations: Software of the Mind (3rd ed.). Columbus, OH: McGraw-Hill.

European Journal of Psychological Assessment (2018), 34(1), 52–63

62

Humrichouse, J., Chmielewski, M., McDade-Montez, E. A., & Watson, D. (2007). Affect assessment through self-report methods. In J. Rottenberg & S. L. Johnson (Eds.), Emotion and psychopathology: Bridging affective and clinical science (pp.

13–34). Washington, DC: American Psychological Association.

Johnson, J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39, 103–129.

Johnson, T., Kulesa, P., Cho, Y. I., & Shavitt, S. (2005). The relation between culture and response styles. Evidence from 19 countries. Journal of Cross-Cultural Psychology, 36, 264–277.

Kay, K., Gathercoal, K., & Buhrow, W. (2011). Does survey format influence self-disclosure on sensitive question items? Computers in Human Behavior, 28, 251–256.

Kazén, M., Kuhl, J., & Quirin, M. (2014). Personality interacts with implicit affect to predict performance in analytic vs. holistic processing. Journal of Personality. Advance online publication. doi: 10.1111/jopy.12100 Krause, S., Back, M. D., Egloff, B., & Schmukle, S. C. (2011).

Reliability of implicit self-esteem measures revisited. European Journal of Personality, 25, 239–251.

Krohne, H. W., Egloff, B., Kohlmann, C.-W., & Tausch, A. (1996).

Untersuchungen mit einer deutschen Version der “Positive and Negative Affect Schedule” (PANAS) [Investigations with a German version of the “Positive and Negative Affect Schedule” (PANAS)]. Diagnostica, 42, 139–156.

Langens, T. A. (2002). Tantalizing fantasies: Positive imagery induces negative mood in individuals high in fear of failure.

Imagination, Cognition, and Personality, 21, 281–292.

Larsen, J. T., McGraw, A. P., & Cacioppo, J. T. (2001). Can people feel happy and sad at the same time? Journal of Personality and Social Psychology, 81, 684–696.

Leventhal, H., & Scherer, K. (1987). The relationship of emotion to cognition: A functional approach to a semantic controversy.

Cognition and Emotion, 1, 3–28.

Mackinnon, A., Jorm, A. F., Christensen, H., Korten, A. E., Jacomb, P. A., & Rodgers, B. (1999). A short form of the positive and negative affect schedule: Evaluation of factorial validity and invariance across demographic variables in a community sample. Personality and Individual Differences, 27, 405–416.

Matthews, G., Jones, D. M., & Chamberlain, A. G. (1990). Refining the measurement of mood: The UWIST Mood Adjective Checklist. British Journal of Psychology, 81, 17–42.

Melvin, G. A., & Molloy, G. N. (2000). Some psychometric properties of the Positive and Negative Affect Schedule among Australian youth. Psychological Reports, 86, 1209–1212.

Merz, E. L., Malcarne, V. L., Roesch, S. C., Ko, C. M., Emerson, M., Roma, V. G., & Sadler, G. R. (2013). Psychometric properties of Positive and Negative Affect Schedule (PANAS) original and short forms in an African American community sample. Journal of Affective Disorders, 151, 942–949.

Muthén, L. K., & Muthén, B. O. (2009). Exploratory factor analysis, confirmatory factor analysis, and structural equation modeling for continuous outcomes. Retrieved from https://www. statmodel.com/download/Topic%201.pdf Muthén, L. K., & Muthén, B. O. (2010). Mplus User’s Guide (Sixth Edition). Los Angeles, CA: Muthén & Muthén.

Nye, C. D., Roberts, B. W., Saucier, G., & Zhou, X. (2008). Testing the measurement equivalence of personality adjective items across cultures. Journal of Research in Personality, 42, 1524–1536.

Oatley, K., & Duncan, E. (1994). The experience of emotions in everyday life. Cognition & Emotion, 8, 369–381.

Osin, N. E. (2012). Izmerenie pozitivnykh i negativnykh emotcii: razrabotka russkoiazychnogo analoga metodiki PANAS [Measuring Positive and Negative Affect: Development of a Russian

European Journal of Psychological Assessment (2018), 34(1), 52–63

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

Version of PANAS]. The Psychology. Journal of Higher School of Economics, 4, 91–110.

Paulhus, D. L., & Vazire, S. (2007). The self-report method. In R. W. Robins, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 224–239).

New York, NY: Guilford.

Payne, B. K., Cheng, C. M., Govorun, O., & Stewart, B. D. (2005). An inkblot for attitudes: Affect misattribution as implicit measurement.

Journal of Personality and Social Psychology, 89, 277–293.

Qiu, L., Zheng, X., & Wang, Y. (2008). Revision of the Positive Affect and Negative Affect Scale. Chinese Journal of Applied Psychology, 14, 249–254.

Quirin, M., & Bode, R. C. (2014). An alternative to self-reports of trait and state affect: The Implicit Positive and Negative Affect Test (IPANAT). European Journal of Psychological Assessment, 30, 231–237.

Quirin, M., Kazén, M., & Kuhl, J. (2009). When nonsense sounds happy or helpless: The Implicit Positive and Negative Affect Test (IPANAT). Journal of Personality and Social Psychology, 97, 500–516.

Quirin, M., Kazén, M., Rohrmann, S., & Kuhl, J. (2009). Implicit but not explicit affectivity predicts circadian and reactive cortisol: Using the Implicit Positive and Negative Affect Test. Journal of Personality, 77, 401–425.

Quirin, M., Kuhl, J., Luckey, U., Pyszczynski, T. A., & Bode, R. (2014). Profound versus superficial coping with mortality threats: Action orientation moderates implicit but not explicit outgroup prejudice. Personality and Social Psychology Bulletin.

Advance online publication. doi: 10.1177/0146167214536928 Robinson, M. D., & Clore, G. L. (2002). Belief and feeling: Evidence for an accessibility model of emotional self-report. Psychological Bulletin, 128, 934–960.

Robles, R., & Páez, F. (2003). Estudio sobre la traducción al español y las propiedades psicométricas de las escalas de afecto positivo y negativo (PANAS) [Study on the spanish translation and the psychometric properties of the Positive and Negative Affect Scales (PANAS)]. Salud Mental, 26, 69–75.

Russell, J. A. (1980). A Circumplex Model of Affect. Journal of Personality and Social Psychology, 39, 1161–1178.

Russell, J. A., Lewicka, M., & Niit, T. (1989). A cross-cultural study of a circumplex model of affect. Journal of Personality and Social Psychology, 57, 848–856.

Schmukle, S. C., Egloff, B., & Burns, L. R. (2002). The relationship between positive and negative affect in the Positive and Negative Affect Schedule (PANAS). Journal of Research in Personality, 36, 463–475.

Schultheiss, O. C., & Brunstein, J. C. (2001). Assessment of implicit motives with a research version of the TAT: Picture profiles, gender differences, and relations to other personality measures. Journal of Personality Assessment, 77, 71–86.

Selcuk, E., Zayas, V., Günaydin, G., Hazan, C., & Kross, E. (2012).

Mental representations of attachment figures facilitate recovery following upsetting autobiographical memory recall. Journal of Personality and Social Psychology, 103, 362–378.

Shields, S. (2002). Speaking from the Heart: Gender and the Social Meaning of Emotion. Cambridge, UK: Cambridge University Press.

Shimoda, S., Okubo, N., Kobayashi, M., Sato, S., & Kitamura, H. (2014). An attempt to construct a Japanese version of the Implicit Positive and Negative Affect Test (IPANAT). Japanese Journal of Psychology, 85, 294–303.

Stieger, S., Voracek, M., & Nader, I. W. (2014). Parent-child proximity: Automatic cognitions matter. Social Indicators Research, 119, 967–978.

Strack, F., & Deutsch, R. (2004). Reflective and Impulsive Determinants of Social Behavior. Personality and Social Psychology Review, 8, 220–247.

Ó 2016 Hogrefe Publishing

M. Quirin et al., A Cross-Cultural Validation of the IPANAT

Terracciano, A., McCrae, R. R., & Costa, P. T. Jr. (2003). Factorial and construct validity of the Italian Positive and Negative Affect Schedule (PANAS). European Journal of Psychological Assessment, 19, 131–141.

Thayer, R. E. (1996). The origin of everyday moods: managing energy, tension, and stress. New York, NY: Oxford University Press.

Thayer, R. E. (2001). Calm energy. New York, NY: Oxford University Press.

Thompson, E. R. (2007). Development and validation of an internationally reliable Short-Form of the Positive and Negative Affect Schedule (PANAS). Journal of Cross-Cultural Psychology, 32, 519–542.

Van Herk, H., Poortinga, Y. H., & Verhallen, T. M. M. (2004).

Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35, 346–350.

Ward, M. K. (2015). Careless responding on internet-based surveys. Observer, 28. Retrieved from http://www. psychologicalscience.org/index.php/publications/observer/2015/ february-15/careless-responding-on-internet-based-surveys. html Ward, M. K., & Pond, S. B. (2013). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554–568.

Watson, D. (2000). Mood and Temperament. New York, NY: Guilford.

Watson, D., & Clark, L. A. (1994). The PANAS-X: Manual for the Positive and Negative Affect Schedule – Expanded Form.

Retrieved from: http://ir.uiowa.edu/cgi/viewcontent.cgi? article=1011&context=psychology_pubs

Ó 2016 Hogrefe Publishing

63

Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54, 1063–1070.

Weinberger, J., Kelner, S., & McClelland, D. C. (1997). The effects of subliminal symbiotic stimulation on free-response and selfreport mood. Journal of Nervous and Mental Disease, 185, 599–605.

Wittenbrink, B., Judd, C. M., & Park, B. (1997). Evidence for racial prejudice at the implicit level and its relationship to questionnaire measures. Journal of Personality and Social Psychology, 72, 262–274.

Yik, M., Russell, J. A., & Steiger, J. H. (2011). A 12-Point circumplex structure of core affect. Emotion, 11, 705–731.

Ziegler, M., & Bensch, D. (2013). Lost in translation: Thoughts regarding the translation of existing psychological measures into other languages. European Journal of Psychological Assessment, 29, 81–83.

Received December 16, 2014 Revision received July 7, 2015 Accepted July 11, 2015 Published online April 22, 2016 Monika Wróbel Institute of Psychology University of Lodz ul. Smugowa 10/12 91-433 Lodz Poland E-mail mwrobel@uni.lodz.pl

European Journal of Psychological Assessment (2018), 34(1), 52–63

Instructions to Authors The main purpose of the European Journal of Psychological Assessment is to present important articles, which provide seminal information on both theoretical and applied developments in this field. Articles reporting the construction of new measures or an advancement of an existing measure are given priority. The journal is directed to practitioners as well as to academicians: The conviction of its editors is that the discipline of psychological assessment should, necessarily and firmly, be attached to the roots of psychological science, while going deeply into all the consequences of its applied, practice-oriented development.

Psychological assessment is experiencing a period of renewal and expansion, attracting more and more attention from both academic and applied psychology, as well as from political, corporate, and social organizations. The EJPA provides a meeting point for this movement, contributing to the scientific development of psychological assessment and to communication between professionals and researchers in Europe and worldwide.

European Journal of Psychological Assessment publishes the following types of articles: Original Articles, Brief Reports, and Multistudy Reports.

Manuscript submission: All manuscripts should in the first instance be submitted electronically at http://www.editorialmanager.com/ejpa. Detailed instructions to authors are provided at http://www.hogrefe.com/j/ejpa Copyright Agreement: By submitting an article, the author confirms and guarantees on behalf of him-/herself and any coauthors that the manuscript has not been submitted or published elsewhere, and that he or she holds all copyright in and titles to the submitted contribution, including any figures, photographs, line drawings, plans, maps, sketches, tables, and electronic supplementary material, and that the article and its contents do not infringe in any way on the rights of third parties. ESM will be published online as received from the author(s) without any conversion, testing, or reformatting. They will not be checked for typographical errors or functionality. The author indemnifies and holds harmless the publisher from any third-party claims.

The author agrees, upon acceptance of the article for publication, to transfer to the publisher the exclusive right to reproduce and distribute the article and its contents, both physically and in nonphysical, electronic, or other form, in the journal to which it has

European Journal of Psychological Assessment (2018), 34(1)

been submitted and in other independent publications, with no limitations on the number of copies or on the form or the extent of distribution. These rights are transferred for the duration of copyright as defined by international law. Furthermore, the author transfers to the publisher the following exclusive rights to the article and its contents: 1. The rights to produce advance copies, reprints, or offprints of the article, in full or in part, to undertake or allow translations into other languages, to distribute other forms or modified versions of the article, and to produce and distribute summaries or abstracts.

2. The rights to microfilm and microfiche editions or similar, to the use of the article and its contents in videotext, teletext, and similar systems, to recordings or reproduction using other media, digital or analog, including electronic, magnetic, and optical media, and in multimedia form, as well as for public broadcasting in radio, television, or other forms of broadcast.

3. The rights to store the article and its content in machinereadable or electronic form on all media (such as computer disks, compact disks, magnetic tape), to store the article and its contents in online databases belonging to the publisher or third parties for viewing or downloading by third parties, and to present or reproduce the article or its contents on visual display screens, monitors, and similar devices, either directly or via data transmission.

4. The rights to reproduce and distribute the article and its contents by all other means, including photomechanical and similar processes (such as photocopying or facsimile), and as part of so-called document delivery services.

5. The right to transfer any or all rights mentioned in this agreement, as well as rights retained by the relevant copyright clearing centers, including royalty rights to third parties.

Online Rights for Journal Articles: If you wish to post the article to your personal or institutional website or to archive it in an institutional or disciplinary repository, please use either a pre-print or a post-print of your manuscript in accordance with the publication release for your article and the document ‘‘Guidelines on sharing and use of articles in Hogrefe journals’’ on the journal’s web page at www.hogrefe.com/j/ejpa.

November 2016

Ó 2018 Hogrefe Publishing

EAPA

APPLICATION FORM EAPA membership includes a free subscription to the European Journal of Psychological Assessment. To apply for membership in the EAPA, please fill out this application form and return it together with your curriculum vitae to: David Gallardo-Puj ol, PhD (EAPAH Secretary Gener al), Dept. of Clinical Psychology & Psychobiology, Campus , Mundet, Pg. de la VallH d H ebron, 171, 08035 B arcelona, Spain, E-mail david.gallardo@ub.edu.

Family name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

First name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Affiliation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

City

. . . . . . . . . . . . . . . .

Postcode . . . . . . . . . . . . . . . . . . . .

Country . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Phone

. . . . . . . . . . . . . . .

Fax . . . . . . . . . . . . . . . . . . . . . .

E-mail

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ANNUAL FEES ◆ EURO 75.00 (US $ 98.00) – Ordinary EAPA members ◆ EURO 50.00 (US $ 65.00) – PhD students ◆ EURO 10.00 (US $ 13.00) – Undergraduate student members

FORM OF PAYMENT ◆ Credit card VISA

Mastercard/Eurocard

IMPORTANT! 3-digit security code in signature field on reverse of card (VISA/Mastercard) or 4 digits on the front (AmEx)

American Express

Number Expiration date

/

CVV2/CVC2/CID#

Card holder’s name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Signature . . . . . . . . . . . . . .

Date

. . . . . . . . . . . . . . . . . . . . .

◆ Cheque or postal order Send a cheque or postal order to the address given above Signature . . . . . . . . . . . . . .

Date

. . . . . . . . . . . . . . . . . . . . .

So that’s how my mind works – Now I get it!

“As a fan of PSI theory for more than 20 years, I am very happy seeing it translated for popular consumption! The book makes the theory reasonably simple, with lots of fun illustrations.”

Kennon M. Sheldon, PhD, Professor of Psychological Sciences, University of Missouri, Columbia, MO

Johannes Storch / Corinne Morgenegg / Maja Storch / Julius Kuhl

Now I Get It!

Understand Yourself and Take Charge of Your Behavior 2018, vi + 248 pp.

US $34.80 / € 27.95 ISBN 978-0-88937-541-3 Also available as eBook Using the example of four colleagues working together in a small company, Now I Get It! shows us the main personality types and their strengths and weakness in such a way that we gain real “now I get it!” insights into what is going on in our own and others’ subconscious.

How does my mind work and what kind of personality do I have? When we can answer these questions and have come to terms with who we are, then the solutions to many issues that arise in everyday life will fall into place. What sort of people do I get on with best and how can I

www.hogrefe.com

best deal with the others? Are there recurring stressful situations in my professional or private life, and how do I resolve them?

This humorously written and illustrated book, by the world’s leading experts in personality systems interaction (PSI) theory and the Zurich Resource Model (ZRM), gives us profound insights into our and other people’s subconscious thoughts – so we can adapt our own behavior and interactions to improve our quality of life. Cartoons and worksheets help us on our way.

Start using strengths today!

“The GO-TO book for building character.”

Martin E. P. Seligman, the founder of positive psychology

Ryan M. Niemiec

Character Strengths Interventions A Field Guide for Practitioners 2018, xx + 300 pp.

US $59.00 / € 46.95 ISBN 978-0-88937-492-8 Also available as eBook This book is the epitome of positive psychology: it takes the “backbone” of positive psychology – character strengths – and builds a substantive bridge between the science and practice. Working with clients’ (and our own) character strengths boosts well-being, fosters resilience, improves relationships, and creates strong, supportive cultures in our practices, classrooms, and organizations. This unique guide brings together the vast experience of the author with the science and the practice of positive psychology in such a way that both new and experienced practitioners will benefit. New practitioners will learn about the core concepts of character and signature strengths and how to fine-tune their approach and troubleshoot. Experienced practitioners will deepen their knowledge about advanced topics such as strengths overuse and collisions, hot button issues, morality, and integrating strengths with savoring,

www.hogrefe.com

flow, and mindfulness. Hands-on practitioner tips throughout the book provide valuable hints on how to take a truly strengths-based approach.

The 24 summary sheets spotlighting each of the universal character strengths are an indispensable resource for client sessions, succinctly summarizing the core features of and research on each strength. 70 evidence-based step-by-step activity handouts can be given to clients to help them develop character strengths awareness and use, increase resilience, set and meet goals, develop positive relationships, and find meaning and engagement in their daily lives.

No matter what kind of practitioner you are, this one-of-a-kind field guide is a goldmine in science-based applications. You’ll be able to immediately bring the science of well-being into action!

A valuable tool for assessing and documenting psychopathology in Now sh!

Engli

Matthew R. Broome / Ronald Bottlender / Michael Rösler / Rolf-Dieter Stieglitz (Editors)

The AMDP System

Manual for Assessment and Documentation of Psychopathology in Psychiatry 9th edition 2018, xii + 198 pp.

US $34.80 / € 27.95 ISBN 978-0-88937-542-0 Also available as eBook Now in its 9th edition, the AMDP System is a widely used tool for documenting psychiatric symptoms in clinical and research projects. It is not only an essential part of many internal clinical documentation systems, but it is also a valuable instrument for training the identification of psychopathological symptoms.

This new edition, now in English, involved the complete revision of the psychopathological and somatic symp-toms, with particular attention to the elimination of ambiguities and inconsistencies, to the precision of definitions, to the exact differentiation of the self and other ratings, as well as to the user friendliness of the system.

www.hogrefe.com

Taking account of developments in recent years in the description and rating of psychopathological symptoms, this latest edition of the AMDP System has been extended by an additional eleven psychopathological and three somatic symptoms and a new section on syndrome formation in the appendix.

This clearly structured manual enables the standardized application of the system, making it an invaluable tool in the training of medicine and psychology students and an essential reference volume in the psychiatric field.

The AMDP System documentation forms are also available separately.