Conceptualizing performance Accred

International Journal for Quality in Health Care 2008; Volume 20, Number 1: pp. 47 – 52 Advance Access Publication: 17 November 2007

10.1093/intqhc/mzm056

Conceptualizing performance in accreditation PERNELLE A. SMITS1, FRANC¸OIS CHAMPAGNE2, DAMIEN CONTANDRIOPOULOS2, CLAUDE SICOTTE2 AND JOHANNE PRE´VAL2 1

PhD Program in Public Health, University of Montre´al, Faculty of Medicine, Montre´al, Que´bec, Canada, and 2GRIS—Groupe de Recherche Interdisciplinaire en Sante´, University of Montre´al, Montre´al, Que´bec, Canada

Abstract Objectives. To compare the conceptualization of performance underlying different accreditation manuals.

Data sources. Accreditation manuals were selected from the 2003 WHO report titled ‘Quality and Accreditation in Healthcare Services’. We used manuals from WHO-listed countries that most influenced the standards: Canada, France, the USA and Australia. The fifth manual is published by the Pan American Health Organization (PAHO).

Extraction methods. Standards from each manual were classified by two independent reviewers. The coding grid, which was based on a Parsonian-based integrative framework on performance, was composed of performance dimensions and their interlinks/alignments.

Principal findings. The four dimensions of quality, goal-attainment, adaptation to the external environment and values, along with their alignments, were given differing levels of importance in the five manuals. The Australian manual emphasizes all four dimensions and their alignments. The PAHO accreditation focuses mainly on quality. The manuals from Canada, France and the USA fall somewhere between the two accreditation extremes of complete versus one-dimensional. Finally, we present a taxonomy of the conceptualization of performance in accreditation manuals that distinguishes between quality-oriented and alignment-oriented accreditation manuals.

Conclusions. Specific conceptualizations of performance underlying accreditation manuals may not be neutral. Perhaps, more normative accreditation manuals are associated with authoritative management styles, or more balanced accreditation manuals with comprehensive management styles. Our comparative analysis is a first step toward better understanding the relationship between the conceptualization of performance and the management style adopted in a particular healthcare organization.

This relationship could help explain the variation observed in healthcare organization performance.

Keywords: accreditation, framework, management, Parsonian perspective, performance

Introduction Accreditation is a procedure that is being used with increasing frequency around the world. It encompasses elements of selfassessment, field survey, reporting [1] and subsequent follow-up [2]. Accreditation is constructed around norms or standards related to the inputs, processes and outputs with which organizations must comply in order to receive accreditation. Originally, the primary goal of healthcare organization accreditation was to improve the performance of health systems through the standardization of practices and quality improvement [3]. It then also became a locus for social change [4]. Since 1990, the number of programs being used worldwide has doubled [5].

Although the accreditation processes of different countries share a common goal of improving the performance of healthcare organizations, each country has developed its own particular solution to achieving this goal.

There has been very little research comparing the content of accreditation manuals since the descriptive work of Se´gouin [6] and the taxonomy proposed by Scrivens [7].

Se´guoin gives an overall description of accreditation processes, including information such as history, cost, length of the accreditation process, the grading system for review visits and the content of accreditation manuals. Scrivens elaborates on the dimensions used in constructing accreditation processes, such as grading schema, voluntary and compulsory participation, the employment status of surveyors, the purposes of accreditation and the focus of standards, but does not provide any details on methodology. Neither Se´guoin nor Scrivens analyze accreditation standards per se.

Accreditation targets the inputs, processes and outputs that an organization is supposed to reach. The level of attainment for each target in accreditation manuals is measured by the organization’s performance. An organization that

Address reprint requests to: Smits Pernelle, Sante´ Publique, University of Montre´al, CP6128 Succ., Centre Ville, 1420 Mont Royal—3rd Floor, Montre´al, Que´bec, Canada H3C 3J7. Tel: þ1 514 343 7365; Fax: þ1 514 343 2207; E-mail: pernelle.smits@ umontreal.ca International Journal for Quality in Health Care vol. 20 no. 1 # The Author 2007. Published by Oxford University Press on behalf of International Society for Quality in Health Care; all rights reserved

47

P. A. Smits et al.

performs well with respect to one target may perform less well with respect to another. Accordingly, a conceptualization of performance focused on a single target or dimension could impair the overall performance of the organization because some areas are underrepresented. Therefore, we have based our analysis of performance on a multidimensional definition of performance. Organizational performance can be defined in different ways depending on the school of thought: the attainment of objectives in the rational goal model, the achievement of quality in the internal process model, adaptation to the external environment in the open model or relationships in the human relation model.

There exist very few models that integrate the different complementary definitions of performance. Indeed, many performance measurement systems have been documented [8, 9]. Some conceptualize performance in business-like terms [10, 11] and exclude staff satisfaction, whereas others are more comprehensive but difficult to operationalize, such as the EFQM Business Excellence Model [8]. Moreover, the EFQM and the Balanced Score Card [12] are aimed at private organizations and do not necessarily incorporate the specificities of public healthcare organizations [13].

For our analysis, we needed a comprehensive model of performance adapted to the healthcare field. We therefore used the Parsonian-based integrative model of performance in healthcare organizations that was developed by Sicotte et al. [14] and used to construct a performance assessment framework of hospitals in Europe [15]. This model has many advantages. First, the complexity of organizational performance is reflected in its definition as a multidimensional concept. Second, the assessment of organizational performance examines more than just the dimensions per se. It takes into account the relationships between these dimensions: what is the level of production reached on the basis of available resources? Is the innovation capacity of the organization appropriate given its stated goals?

The concept of performance needs to include the relationships between dimensions. We believe the model of performance developed by Sicotte is well-suited to evaluating organizational performance in all its complexity.

The aim of this paper is to analyze the concept of performance underlying five recognized accreditation manuals from the countries of Australia, the USA, France, Canada and from the Pan American Health Organization (PAHO).

We examine which dimensions of performance were included in these accreditations (manual) through a descriptive analysis of the manuals. After selecting the manuals, we analyzed the available ones using the integrative framework on healthcare organization performance. We classified the standards of each accreditation manual according to the framework’s dimensions of performance.

Methodology Selection of accreditations We used a quantitative method for pre-selecting recognized accreditations based on the volume of citation for each

48

WHO-listed country that most influenced the selection of standards (WHO report) [16]. We counted the frequency of citation of program X listed as influential (frequency ¼ number of times program X is cited/total number of answers from responding agencies). The results for frequency of citation were 34% for the US program, 24% for the Canadian, 24% for the Australian, 9% for the English and 3% for the Japanese and Polish.

For our in-depth analysis, we selected the most commonly cited programs for which a hospital accreditation manual was available to us. We were not able to analyze the accreditation manual used in Great Britain, as we had no version of it and it was not available to the public. The Japanese manual was also excluded, as a translated version was not available. The French manual was included to extend the analysis to a European country. The PAHO manual was included to extend the analysis to low- and middle-income countries. The final accreditations retained for our analysis are as follows, by country/organization: (i) Canada: Achieving Improved Measurement Program, Canadian Council on Health Services Accreditation [17]. (ii) USA: Comprehensive Accreditation Manual for Hospitals: The Official Handbook, Joint Commission on Accreditation of Healthcare Organizations [18]. (iii) France: Accreditation for Healthcare Organizations: Second Accreditation Procedure, Agence Nationale d’accre´ditation et d’e´valuation en sante´ [19]. (iv) Australia: EQuIP Guide of the Australian Council on Healthcare Standards (3rd edn) [20]. (v) PAHO: La garantia de calidad [21], which we translated to the best of our ability.

Please note that, in this paper, we refer to the manuals by country (except in the case of the PAHO manual) for the sake of simplicity. We recognize that more than one accreditation standard may be used in a country (for example, the JCAHO and NCQA standards in the USA).

Selection of the framework of analysis We rely on a Parsonian description of action and its application to the concept of performance, as it applies to the specific case of healthcare organizations [14]. This model (Fig. 1) was an attempt to synthesize the common core elements addressed in the healthcare organization performance literature. It is based on the idea of achieving an equilibrium among four dimensions of performance: (i) adaptation (A), the organization’s capacity to survive and grow in the changing environment; (ii) goals (G), the results pursued in terms of efficiency, effectiveness, the attainment of outcomes and stakeholder satisfaction; (iii) integration (I) or production, the care and services produced by the healthcare organization in terms of volume of care and mechanisms and (iv) latency (L) or values and culture, the sense-making in the organization and its social environment. Values and culture refer to organizational climate, resolution of staff conflicts, rewarding system and staff motivation. These four dimensions are related to each other through six inter-linked systems or alignments [14].

Conceptualizing performance in accreditation

Results Representation of the dimensions of performance and the alignment between dimensions in the accreditations

Figure 1 Representation of a conceptual framework for the analysis of healthcare organization performance (adapted from Sicotte et al. [14]).

This framework will be used to order the diverse standards of accreditation and to provide the analytical groundwork for our comparison of accreditations and our analysis of the conceptualization of performance underlying the five recognized accreditations.

Analysis of the standards in the selected accreditations Two persons independently read the selected manuals. Using the integrative framework, they both independently classified the standards on the basis of their understanding of the manual’s content with respect to each dimension and alignments of the Sicotte framework of analysis. After this classification step, the two persons discussed any differences in order to arrive at one, mutually agreed upon classification of the standards. One person did a comparative analysis of the most versus least common dimensions and alignments used.

This comparison was based on the number of standards categorized in each dimension of the framework of analysis.

It produced a taxonomy of accreditation manuals on the basis of their conceptualization of performance.

Table 1 shows the importance of each dimension of performance as defined by the integrative model [14] for each of the five accreditation manuals. The more the plus (þ) signs, the more fully represented a dimension in the accreditation. A plus/minus (þ/2 ) sign signifies a quasi-absence of standards for that particular dimension, whereas a minus (2) sign signifies a complete absence of standards.

Table 2 shows the presence (P) or absence (A) of alignments.

Results for the USA. The US accreditation manual [18] emphasizes the dimension of integration/production (Table 1): ‘the hospital has a process to ensure that a person’s qualifications are consistent with his/her job responsibilities’ (from the section titled ‘Management of Human Resources’; falls under the subdimension of quality in Fig. 1). Less emphasis is given to the dimensions of adaptation to the environment, goal attainment and values.

Tactical and allocation alignments are both present (Table 2) to highlight the importance of linking goals to the production of services and of relating adaptation to the production of services.

Results for PAHO. Using the same framework to categorize the standards used, we found that the PAHO accreditation [21] omits goals and values. There is a considerable focus on the production of services, especially the quality aspect and a lesser focus on adaptation to the environment (Table 1).

However, goals are taken into account through their alignment with adaptation and with production (Table 2).

Results for France. The French accreditation [19] places little emphasis on values, stressing instead the importance of quality and, to a lesser extent, adaptation and goals (Table 1).

There is some linking between adaptation and goals through strategic alignment, and between adaptation and production through allocation alignment (Table 2).

Results for Canada. The Canadian accreditation [17] gives similar importance to adaptation, production and values. It places somewhat less importance on goals (Table 1). There is some alignment of values and adaptation, of values and goals and of adaptation and goals (Table 2).

Table 1 The importance of the dimensions of performance in the five selected accreditation manuals Integrated framework

USA

PAHO

France

Canada

Australia

.............................................................................................................................................................................

A G I L

Present Present Present Present

þþ þ þþ þ þ

þ/2 – þþ þ –

þ þ þþ þ þ/2

þþ þ þþ þþ

þ 2 þþ þ/2

Each dimension is either absent (2) or quasi-absent (þ/2 ) or more and more present (þ to þþþ ).

49

P. A. Smits et al.

Table 2 The presence/absence of alignments in the five selected accreditation manuals Integrated USA PAHO France Canada Australia framework ....................................................................................

SA TA OA CA AA LA

P P P P P P

A P A A P A

P P A A A A

P A A A P A

P A A P A P

P P P P P P

P, presence of the alignment; A, absence of the alignment; SA, strategic alignment; TA, tactical alignment; OA, operational alignment; CA, contextual alignment; LA, legitimization alignment; AA, allocative alignment.

Results for Australia. The Australian accreditation [20] makes reference to the dimensions of adaptation and quality, although making almost no mention of values and completely omitting goal attainment (Table 1). However, goals and values are not totally absent, as they are linked to the other four dimensions through the presence of all six alignments (Table 2).

To summarize, all of the selected accreditations emphasize the production of services, especially quality and coordination of services and adaptation to the environment. Some omit values or goal attainment. The dimensions of performance, when not mentioned directly, are included in relation to other dimensions through the alignments. For example, the attainment of goals is assessed by several means: a standard of accreditation can be specific to the dimension of goals, as it is the case in the manuals from France, the USA and Canada (i.e. ‘human resources achieving positive outcomes’ [17]), or it can be linked to the alignment of goals with the other dimensions, as it is the case in the PAHO and Australian manuals (i.e. ‘management of human resources supports the delivery of quality, safe care, and services/ human resources planning supports the organization’s current and future ability to provide quality, safe care and services’ [20]).

Taxonomy of accreditations We propose a taxonomy of the standards of accreditation manuals to compare them with respect to their conceptualization of performance. We chose to classify the various manuals using two axes reflecting the importance they give to both the individual dimensions and to the alignments between dimensions.

The results given earlier show that quality is the only dimension present in every accreditation manual (Table 1).

Moreover, accreditations vary in how much emphasis they place on production, especially its quality and on the alignments between dimensions. Therefore, we decided to organize the taxonomy around these two axes: normative

50

Figure 2 Visual representation of a taxonomy of accreditation based on the conceptualization of performance.

The taxonomy is based on two continuums: balanced accreditation on the horizontal axis and normative accreditation on the vertical axis. An accreditation manual could occupy a high (þ) or low (2) position on each continuum.

quality-oriented axis and balanced alignment-oriented axis. In Fig. 2, one axis represents a quality-oriented accreditation, ranging from normative to non-normative accreditation, and the other represents an alignment-oriented axis ranging from more (þ) to less (2) balanced accreditation.

A normative accreditation is one that places a relatively high emphasis on the dimension of production, especially on the subdimension of quality, compared with the emphasis placed on the other three dimensions. The accreditation manuals of PAHO, France and the USA are highly normative accreditations that focus on production. Those from Canada and Australia, which are positioned at the bottom of Fig. 2, are less normative accreditations since they are less focused on quality.

A balanced accreditation is one that places great emphasis on alignments (Table 2). We hypothesize that accreditation manuals composed of many alignments will achieve higher organizational performance. On the basis of the horizontal position along the axis representing degree of alignment between dimensions, the most balanced alignment-oriented accreditation manual is the one from Australia, less balanced are the manuals from the PAHO, France, Canada and the USA. Ideally, accreditation should be balanced and take quality into account. We would expect balanced accreditation manuals on the right-hand side of the diagram to achieve higher performance than accreditation manuals positioned on the left-hand side. Whether accreditation manuals situated in the top right-hand corner will lead to higher performance than those situated in the bottom right-hand corner or vice versa remains to be seen. We hypothesize that the omission of normative dimensions, as represented by the bottom half of the diagram, is not favorable. However, accreditation manuals positioned in the top half of the diagram and therefore emphasizing normative procedures could also be associated with negative outcomes. We recommend that more

Conceptualizing performance in accreditation

empirical studies on the actual performance of accreditations positioned at both extremes—top and bottom halves of the diagram—be carried out in order to determine the ideal position for an accreditation in the taxonomy.

Discussion To improve the performance of healthcare organizations, we need to have a clear understanding of the definition of performance. Indeed, the definition of performance does not refer to just one, mutually agreed-upon dimension. The integrative framework we chose allows us to classify and analyze accreditations according to a number of different dimensions.

We found that some dimensions were common across all accreditations studied—e.g., the production of services.

Some, e.g., values, are scarcely mentioned in any accreditations. The most common standard across all accreditations was quality, reflecting a normative view of the performance of healthcare organizations. The least common dimension was ‘values and culture’. The other dimensions of goal attainment and adaptation to the environment were emphasized to varying degrees in the different manuals.

The Parsonian-based perspective of performance presents four dimensions that healthcare organizations have to fulfill in order to achieve high performance levels. Therefore, we can theoretically hypothesize that an accreditation built around all four dimensions produces the highest level of performance. In this respect, on the basis of our framework of analysis, the Canadian manual scores above the others. The US manual occupies an intermediate position, followed by the French and Australian manuals. The PAHO manual is the least complete.

An organization that performs well with respect to one criterion may perform less well with respect to one or more other criteria [22]. A conceptualization of performance focused solely on one criterion or dimension could impair the organization’s overall performance because of the poor emphasis placed on other important areas. We can therefore hypothesize that accreditations that include alignments between the dimensions will lead to better performance. The Australian accreditation manual appears to be the most complete, balanced manual since it contains all of the alignments of our integrative model. The Canadian manual appears to be less complete, and the French, US and PAHO manuals are the least complete. On the basis of our analysis of the conceptualization of performance underlying accreditation manuals, we were able to create a taxonomy of accreditation manuals reflecting the underlying concepts of performance based on two axes: normative quality-oriented accreditation and balanced alignment-oriented accreditation.

The results of this research apply to the particular manuals studied and are not a static description of a country’s vision of healthcare organization performance. The PAHO manual that we analyzed dates from 1992, whereas the other manuals date from the past six years. Countries modify their manuals every 3 – 5 years. The various Hispanic countries may have devised different protocols to assess

management and so may not be using the one presented in the PAHO manual. Another limit of our analysis relates to the weighting applied by accreditation agencies to particular important standards. We did not take such weighting into consideration in our analysis. Finally, the results may have been different had we chosen a different framework of analysis to classify accreditation standards.

Our comparative analysis of the diversity of performance conceptualizations parallels research into how the conception of management varies across countries [23]. Our analysis can be a useful starting point for further studies on the relationship between management style and performance in healthcare organizations. One interesting avenue would be to compare the management style of an organization with its performance conceptualization model as identified by the typology.

To date, no studies have empirically evaluated the extent to which the dimensions and alignments considered in an accreditation manual improve organizational performance. Such research would provide valuable information that could be used to improve the performance of our healthcare organizations.

Specific conceptualizations of performance underlying accreditation manuals may not be neutral. More normative accreditation manuals may be associated with authoritative management styles, and more balanced accreditation manuals with comprehensive management styles. Our comparative analysis is a first step toward better understanding the relationship between the conceptualization of performance and the management style adopted in a particular healthcare organization.

Acknowledgements We would like to thank Myriam Hivon and Susan Lemprie`re for useful comments, and two anonymous reviewers for their constructive criticism and suggestions.

Funding GETOS Chair at the Universite´ de Montre´al supported this work financially.

References 1. Hayes J, Shaw C. Implementing accreditation systems. Int J Qual Health Care 1995;7:165 –71.

2. Jovanovic B. Hospital accreditation as method for assessing quality in healthcare. Arch Oncol 2005;13:156– 7.

3. Robert MD, James S, Jack G et al. A history of the joint commission on accreditation of hospitals. J Am Med Assoc 1987;258:21.

4. Pomey M, Contandriopoulos AP, Franc¸ois P et al. A tool for organizational change in hospitals? Int J Health Care Qual Assur 2004;7:113 –24.

5. Shaw CD. Editorial: evaluating accreditation. Int J Qual Health Care 2003;15:455–6.

51

P. A. Smits et al.

6. Se´gouin C. L’accre´ditation des ´etablissements de sante´. De l’expe´rience internationale a` l’application franc¸aise. Paris: Douin Editeurs, 1998.

for Europe PATH 2005;17:487–96.

project.

Int

J

Qual

Health

Care

7. Scrivens EA. A taxonomy of the dimensions of accreditation systems. Soc Policy Admin 1996;30:114–24.

16. WHO. Quality and Accreditation in Health Care Services: A Global Review. World Health Organization, 2003.

8. Neely A, Mills J, Platts K et al. Performance measurement system design: developing and testing a process-based approach. Int J Oper Prod Manage 2000;20:1119 –45.

17. CCHSA. AIM: Achieving Improved Measurement: Accreditation Program. Ottawa: Canadian Council on Health Services Accreditation, 2001.

9. Tangen S. Performance measurement: from philosophy to practice. Int J Prod Perform Manage 2004;53:726– 37.

18. JCAHO. Comprehensive Accreditation Manual for Hospitals: The Official Handbook (CAMH). Joint Commission on Accreditation of Healthcare Organizations, 2006.

10. Cross KF, Lynch RL. For good measure. CMA Magazine, April 1992;20 –3.

11. Sink D, Tuttle T. Planning and Measurement in Your Organisation of the Future. Norcross, GA: Industrial Engineering and Management Press, 1989.

12. Kaplan R, Norton DP. The balanced scorecard—measures that drive performance. Harv Bus Rev 1992;10:71–9.

13. Holmes JS, Amin Gutie´rrez de Pin˜eres S, Kiel DL. Reforming public administration: applying the balanced scorecard approach to government agencies in developed and developing countries.

Int J Public Admin 2006;29:1125 –45.

14. Sicotte C, Champagne F, Contandriopoulos AP et al.

Conceptual framework for the analysis of health care organizations’ performance. Health Serv Manage Res 1998;11:24–48.

15. Veillard J, Champagne F, Klazinga N et al. A performance assessment framework for hospitals: the WHO regional office

52

19. ANAES. Accreditation for Healthcare Organizations: Second Accreditation Procedure, 2004. http://www.has-sante.fr/portail/ types/FileDocument/doXiti.jsp?id=c_269762 (20 August 2007, date last accessed).

20. ACHS. The EQuIP Guide: A Framework to Improve Quality and Safety of Health Care. 3rd edn. Australia: The Australian Council on Healthcare Standards, 2002.

21. PAHO. La garantia de calidad. Acreditacion de hospitales para america latina y el caribe. Pan American Health Organization, 1992.

22. Cameron K. Effectiveness as paradox: consensus and conflict in conceptions of organizational effectiveness. Manage Sci 1986;32:539 –53.

23. Pindur W, Rogers SE, Kim PS. The history of management: a global perspective. J Manage Hist 1995;1:59 –77.

Accepted for publication 17 October 2007