SHS Vol 2, No 2 (2017)

Incorporating the Southern African Journal of Public Health NOVEMBER 2017 | VOLUME 2 | ISSUE 2

Capacity building for strengthening health systems in Africa A hospital-management training programme in South Africa

EDITOR IN CHIEF Debashis Basu

NOVEMBER 2017 | VOLUME 2 | ISSUE 2 Official journal of the South African Public Health Medicine Association, an affiliate of the South African Medical Association

EDITORIAL BOARD Leegail Adonis Michael Asuzu Chauntelle I Bagwandeen Lilian Dudley Francis Hyera Willem Kruger Shan Naidoo Petrus G Rautenbach PUBLISHED BY THE HEALTH AND MEDICAL PUBLISHING GROUP CEO AND PUBLISHER Hannah Kikaya HannahK@hmpg.co.za

EDITORIAL 25

Professionalisation of the training of healthcare managers: Where are we?

D Basu

ARTICLES 26

Challenges and limitations to adopting health-technology assessment in the South African context D Croce, D Mueller, D Tivey

30

Addressing healthcare-management capacity building: The story of the African Institute for Healthcare Management S M Sammut

34

A hospital-management training programme in South Africa S Naidoo, M Mothagae, B Kistnasamy, C Jinabhai

CHIEF OPERATING OFFICER Diane Smith 012 481 2069 dianes@hmpg.co.za EXECUTIVE EDITOR Bridget Farham PRODUCTION MANAGER Emma Jane Couzens MANAGING EDITORS Claudia Naidu Naadia van der Bergh ONLINE SUPPORT Gertrude Fani TECHNICAL EDITOR Kirsten Morreira

PUBLIC HEALTH NOTEBOOK

40 Parametric hypothesis tests for the difference between two population means

L N Dzikiti, B V Girdler-Brown

DTP & DESIGN Clinton Griffin Travis Arendse HEAD OFFICE Health and Medical Publishing Group (Pty) Ltd Block F, Castle Walk Corporate Park, Nossob Street, Erasmuskloof Ext. 3, Pretoria, 0181 Tel. 012 481 2069 Email: dianes@hmpg.co.za CONTACTS Website and online submissions: www.shsjournal.org.za Publisher website: www.hmpg.co.za Editorial queries: shseditor@hmpg.co.za Tel: +27 (0)21 532 1281 ISSN 2312-9360

Published by the Health and Medical Publishing Group (Pty) Ltd Company registration 2004/022032/07, a subsidiary of the South African Medical Association Suite 11 Lonsdale Building, Gardener Way, Pinelands, 7405 Tel: +27 (0)21 532 1281 | Email: publishing@hmpg.co.za Creative Commons Attribution - Non Commercial Works License (CC BY-NC 4.0)

This journal, Strengthening Health Systems, is made possible by the generous support of the American people through the United States Agency for International Development (USAID) and the President’s Emergency Plan for AIDS Relief. The contents are the responsibility of Strengthening Health Systems and do not necessarily reflect the views of USAID or the United States Government

This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0.

EDITORIAL

Professionalisation of the training of healthcare managers: Where are we?

The health workforce is one of the six building blocks of a health system.[1] The World Health Organization (WHO) has developed a new global strategy for human resources for health, which defines four core objectives around evidence-based policies, the alignment of investments in human resources for health with current and future population needs, the capacity of institutions for action, and accountability.[2] Capacity building remains one of the priorities for the optimisation of human resources. Target 17.9 of the Sustainable Development Goals is dedicated to capacity building, and aims to ‘enhance international support for implementing effective and targeted capacity building in developing countries to support national plans to implement all the sustainable development goals, including through North-South, South-South and triangular co-operation’.[2] However, the vast majority of health-system capacity-building efforts have focused on enhancing medical and public health skills; less attention has been directed at developing healthcare managers, despite their central role in improving the functioning and quality of healthcare systems.[3] In South Africa (SA), several initiatives have been undertaken to build capacity among healthcare managers over the last two decades. In 2012, a new Institute for Leadership and Management in Health was launched, and an advisory committee was formed to that effect, as part of the WHO’s Global Strategy on Human Resources for Health: Workforce 2030.[4] Recently, the SA National Department of Health published a notification inviting nominations for the newly

25

November 2017

established National Governing Body on Human Resources of Health (incorporating training and development).

Sammut[5] presents the unique concept of the establishment of an African Institute for Healthcare Management, based on his experiences at Strathmore Business School and elsewhere. Naidoo et al.[6] share their experiences of training public-sector hospital managers through a formal Master’s programme.

Hopefully, these capacity-building initiatives in SA and elsewhere will help us to professionalise the training of health managers, and assist in the effective and efficient implementation of National Health Insurance.

Deb Basu Editor Strengthen Health Syst 2017;2(2):25. DOI:10.7196/SHS.2017.v2.21.62 1. World Health Organization. Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes – WHO’s framework for action. Geneva: WHO, 2007. http://www.who.int/healthsystems/ strategy/everybodys_business.pdf (accessed 31 August 2017).

2. World Health Organization. Global Strategy on Human Resources for Health: Workforce 2030. Geneva: WHO, 2016. http://who.int/hrh/resources/globstrathrh-2030/en/ (accessed 31 August 2017).

3. Kebede S, Abebe Y, Wolde M, Bekele B, Mantopoulos J, Bradley EH. Educating leaders in hospital management: A new model in sub-Saharan Africa. Int J Qual Health Care 2010;22(1):39-43. https:// doi.org/10.1093/intqhc/mzp051 4. National Department of Health, South Africa. Human Resources for Health in South Africa. Pretoria: NDoH, 2011. https://www.gov.za/sites/default/files/hrh_strategy_0.pdf (accessed 31 August 2017).

5. Sammut S. Addressing healthcare management capacity building: The story of AIHM. Strengthen Health Syst 2017;2(2):28-31. https://doi.org/SHS.2017.v2.i2.56 6. Naidoo S, Mothoagae M, Kistnasamy B, Jinabhai C, Basu D. A hospital management training programme in South Africa. Strengthen Health Syst 2017;2(2):32-37. https://doi.org/10.7196/ SHS.2017.v2.i2.63

This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0.

ARTICLE

Challenges and limitations to adopting healthtechnology assessment in the South African context D Croce,1 BEng, MBA; D Mueller,2 MEng; D Tivey,3 PhD entro di Ricerca in Economia e Management in Sanità e nel Sociale, Faculty of Management Engineering, Università Carlo Cattaneo (LIUC), C Castellanza, Italy 2 Charlotte Maxeke Medical Research Cluster (CMeRC), University of the Witwatersrand, Johannesburg, South Africa 3 Research and Evaluation, incorporating ASERNIP-S (Australian Safety and Efficacy Register of New Interventional Procedures - Surgical) , Royal Australasian College of Surgeons, Adelaide, Australia 1

Corresponding author: D Croce (dcroce@liuc.it)

Healthcare systems across nations develop in response to the social, economic and political situations of each country. Nevertheless, all health systems cater to the health needs and wellbeing of their population. This is achieved through both the delivery and financing of services to ensure patient access to health services and technologies. However, all countries, no matter whether developed or developing, are faced with finite resources. To achieve the maximum health benefits for the public, decisions need to be made on the organisation of the health-service delivery system, the type of interventions to be offered and the manner of service delivery. There are two distinct parts to a decision-making process: the collection and synthesis of evidence, as part of a health-technology assessment (HTA), and the appraisal of the evidence, framing the recommendations to a given context. HTA is a multidisciplinary study that provides information on the introduction and diffusion of the technology in question, as well as information on price, reimbursement and the appropriate targeting of the technology for effective clinical practice. There are various methods and frameworks available to conduct HTAs. Here, we showcase the European network for Health Technology Assessment (EUnetHTA) Core Model, developed in Europe, that is used fully, partially or adaptively

by European countries. As decision-making is the process of selection of the best possible course of action among all the available options, those making decisions need tools to consider multiple factors in a consistent, transparent and reproducible manner, for example, by using multiple-criteria decision analysis. However, it is important to consider that these tools were developed in countries with health systems very different to those of South Africa and low- to middle-income nations, which raises the question of their limitations and the adaptations necessary to fit to the health-system needs of these countries.

Strengthen Health Syst 2017;2(2):26-29. DOI:10.7196/SHS.2017.v2.i2.58

Irrespective of country, policy-makers see the need to govern and manage an ever-expanding diffusion of new and innovative health technologies in national health services.[1] A health technology may, for example, be a drug, an organisation of services, a medical device or a diagnostic. This demand has led to the advancement of health-technology assessment (HTA) techniques to generate ‘fit-forpurpose’ reports that present the best available evidence to inform policy-making. HTA is the study of the medical, social, organisational, ethical and economic implications of the development, diffusion and use of health technology. It is a multidisciplinary field of policy analysis, and encompasses the assessment of the quality, safety, efficacy, effectiveness and cost-effectiveness of healthcare interventions and technology. Therefore, tools such as costeffectiveness analysis and budget-impact models, although still widely used, are not adequate for evaluating a health technology, as their one-dimensional nature is insufficient to capture the typical multiple benefits of a health technology.[2] The first steps to document HTA methodologies can be traced back to the

seminal work titled Canadian Economic Evaluation Guidelines (1st ed.

November 1994),[3] published by the Canadian Agency for Drugs and Technologies in Health. This was followed by a work by Goodman[4] in the USA, the latest online edition of which was updated in 2014, and one by the Israeli Center for Technology Assessment in Health Care in 1998.[5] This level of rigour has subsequently been applied by other disciplines, especially in the fields of clinical science, life sciences and engineering science, with a focus on the development of HTA methods that incorporate all aspects of healthcare delivery and health economics. The outcome of these innovations in HTA has been a remarkable increase in the number of HTA agencies and the volume of assessments.

The EUnetHTA Core Model – a standardised approach to HTA This proliferation of HTA agencies and the consequent multitude of approaches to HTA resulted in the need for the development of common methodologies, to increase the transferability of HTA

November 2017

26

ARTICLE

The challenges of introducing HTA Using Italy as a case study, what were the challenges that needed to be addressed in adopting HTA and the EUnetHTA Core Model for decision-making and policy development? In Italy, the first steps

Health problem and current use

Health Technology Assessment Core Model

Description and technical characteristics Safety Clinical effectiveness Costs and economic evaluation Ethical analysis Organisational aspects Social aspects Legal aspects

Fig. 1. Domains of the EUnetHTA Core Model.

27

November 2017

AE

AE = core assessment element

AE

AE = non-core assessment element

INFO

INFO = Locally produced information that does not follow HTA Core Model

Pool of structured health technology assessment (HTA) information Domain 1

HTA Core Model

reports across jurisdictions. This paved the way for the evolution of the European model, the European network for Health Technology Assessment (EUnetHTA) Core Model,[6] which is now in its third iteration. EUnetHTA (http://www.eunethta.eu/about-us) is a network of HTA organisations across Europe, established with the aim of standardising evidence generation and working jointly towards providing support to policymakers at national and regional levels. To accomplish this task, the Core Model was developed by EUnetHTA members for its members. The model provides a standard method for evidence synthesis, structured and presented in a standard format.[7] The model is divided into nine domains (Fig. 1) representing the use, consequences and implications of the health technology in question. A domain is divided into several topics, and each topic is further divided into several issues. Each issue covers a specific area and is divided into separate questions, the responses to which provide an assessment of a health intervention.[7] An assessment element is a combination of a domain, topics and issues (Fig. 2). A by-product of this development is a simplified rapid-HTA model[8] that is currently being used in various countries.

Domain 2

AE

AE

AE

AE

AE

AE

AE

AE

AE

Local report

Core HTA AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

Rapid HTA Domain 3

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE AE

AE

AE

INFO

INFO

AE

INFO

AE AE

Relative effectiveness assessment Domain 4

AE

AE

AE

AE

AE

Takes into account local epidemiology, resources, values, etc.

Fig. 2. EUnetHTA Core Model implementation scheme (adapted from the presentation by Finn Børlum Kristensen, Director, Secretariat for EUnetHTA, Croatia meeting, 2010). (HTA = health-technology assessment).

towards HTA can be traced back to the initiatives of the National Institute of Health in the 1980s, when the effectiveness and safety of large equipment was considered from a technical viewpoint.[9] This was followed by the introduction of the National Health Plan in 2006, and the founding of the National Agency for Regional Health Services (AGENAS), a technical scientific agency which oversees the uniformity of services offered to citizens by various regional health services. To date, there has been limited uptake of HTA, mainly due to the fragmentation of the 21 regional health services. What can be learnt from the Italian experience? In brief, the factors that hinder the introduction and further development of the HTA methodology, the sole purpose of which is to serve the policy- and decision-makers,[10] can be classified as being organisational, scientific and material.[11] These limitations need to be assessed and addressed if the introduction of HTA in Africa is to be a success.

Organisational challenges reside in developing an awareness in policy- and decision-makers that HTA exists, and then educating them on the strengths and weaknesses of the tool. Conducting HTA, and the use of decision-making tools such as multiple-criteria decision analysis (MCDA), are not alternatives to making decisions about where healthcare funds

are spent. Our experience shows that there is a misconception that HTA is a substitute for the decision-making process, rather than a tool to assist in making decisions, and this needs to be overcome for the effective introduction and use of HTA. In addition, another major organisational issue is capacity building to develop and retain a core group of experts capable of conducting HTA. There are abundant training courses; however, there are few opportunities for the mentoring of staff to adopt HTA within the context of their health service. The International Network of Agencies for HTA (http://www.inahta.org/), through its mentorship programme, provides a potential avenue to gain the necessary experience. These programmes are being supplemented by emerging postgraduate qualifications that are grounded in the practical application of skills for technology assessments – for example, the HTA programme at the University of Pretoria.

Capacity building aside, another common issue facing new HTA agencies in the low- and middle-income countries (LMIC) setting is staff retention. This was highlighted by attendees at workshops at the Rome meeting of the Health Technology Assessment International Society[12] and that conducted for the National Department of Health (South Africa, SA) held in Pretoria (24 - 25 November 2017). Attendees highlighted

ARTICLE the fact that in many LMIC settings, the HTA agencies are seen as training platforms for subsequent employment in the medical device and pharmaceutical industries, with the primary driver for this movement being salary considerations. Organisationally, the aspect of staff retention needs to be addressed through competitive salaries and clear career pathways.[13] The scientific limitations reside in the wide variety of HTA methodologies, the complexity of the methods and the absence of an overall judgment (of ‘accept’ or ‘do not accept’) being made at the end of the assessment process. Variation in method is overcome by adopting the EUnetHTA Core Model. However, the model is very rigorous, and it requires significant resources to complete the lengthy and complicated process of preparing a full HTA report. This left room for the creation and use of simplified HTA templates, from the Danish mini-HTA[14] to various hospital-based models.[15] Given the rigour of the HTA Core Model, and the detailed guidance it proffers regarding how to identify the best available evidence for a given technology, there was a move to develop a simplified rapid-HTA model which is based on the Core Model.[8] This has now been achieved, and the rapid method is currently being used across Europe. For those new to HTA, the rapidHTA method that is built on the solid foundation of the Core Model 3 provides the novice practitioner of HTA or a new HTA agency with a clear methodology to follow. Importantly, adopting this standardised approach allows benchmarking against expert practitioners and established agencies.

Restrictions on the use of HTA To date, the application of different HTA methodologies, which are often applied with varying degrees of pragmatism, has been problematic,[16] as the variation in methods makes a comparison of the HTAs carried out on the same health technology extremely difficult. This is further complicated by variation in health services and social context, and the underlying burden of disease within a given population. Adopting a reference model, as happened across Europe with EUnetHTA’s Core Model 3, does address some of these restrictions based on methodology. Similarly, it is difficult to fully apply the results of a HTA across different models in health services (e.g. universal v. mutual), and in different regions (e.g. Africa v.

Europe). Aside from different regional contexts, different healthcare payment systems for performance in the delivery of care create an additional barrier to adopting existing HTA reports.[17] It seems that these barriers to the applicability of HTA reports across jurisdictions make it counterintuitive to proceed with a single model. However, it is only through the adoption of common methodologies that the potential for reuse of existing HTA reports can be considered and achieved. Certainly, the use of common methodologies will assist HTA practitioners in determining what is common and relevant to their setting, to provide shortcuts for rapid assessment and ensure that evidence is representative, while limiting bias to a minimum.

We suggest that the EUnetHTA Core Model 3 be used as the reference standard for developing a variant of HTA that is Africacentric. The Core Model is complicated (9 domains, 134 elements),

as it tries to capture all the nuances between different dimensions, different points of view and different actors (it is multiprofessional).

To reduce the number of elements of the model would require a careful appraisal of the domains and assessment elements to eliminate those that are not relevant to the African context, and thereby give greater weight, sensitivity and depth of knowledge to the elements that are relevant. The challenge and potential restriction to adopting HTA within Africa is to adapt the Core Model in a non-arbitrary manner.

Theoretically, this approach is promoted by EUnetHTA in Fig. 2.

A major goal of the collaboration between the Charlotte Maxeke Medical Research Cluster (CMeRC, SA), the Australian Safety and Efficacy Register of New Interventional Procedures – Surgical (ASERNIP-S) and Università Carlo Cattaneo (LIUC, Italy) is to refine the rapid version of the EUnetHTA Core Model 3 to be contextually relevant to SA, as well as to other African countries.

Finally, a quantitative approach, such as MCDA, could support the use of the HTA-based decision model. The introduction of a healthcare technology is based on risk-benefit evaluation.[18] MCDA is applied to decisions with multiple objectives, and can be used to appraise different alternatives with conflicting criteria.[19] MCDA techniques need to be carefully constructed, with appropriate training of users. The process requires decisions to be broken into domains, and the application of the EUnetHTA model, i.e. its nine domains. The participants in the MCDA process must weigh the importance of each domain against the technology being analysed, for example, pathologies with many therapeutic alternatives compared with pathologies with few or no therapeutic alternatives. Then, based on the best available evidence as presented in HTA reports, each domain is judged and given a numerical value that is adjusted by the assigned weight for each domain as pertaining to the technology under review.

Adopting tools such as MCDA applies a similar level of rigour, transparency and reproducibility to the use of HTA reports as to that applied to their generation.

Conclusion Countries with limited resources, such as those in Africa, have been challenged by the continuous influx of new and innovative health technology, whether voluntarily or involuntarily. Policyand decision-makers are in need of reliable, timely, transparent and rigorous information to support their decisions on the prioritisation, selection, utilisation, diffusion and disinvestment of health interventions, leading to better outcomes for patients and the public. Equity among citizens is still a very important value.

Standardised and detailed methodologies such as the EUnetHTA Core Model provide a framework for collecting, synthesising and sharing information. As stated, decision-makers are faced with multiple alternatives and multiple factors influencing these decisions, which situation requires a robust and quantitative approach to the appraisal process, as provided by the MCDA tool.

It is to be noted that the authors are working on the adaptation of the model to the SA context, to enable the production of reports suitable for the context.

As to the limitations on the uses of HTA methodology, perhaps

November 2017

28

ARTICLE the main limitation lies within governance at the meso-level of the healthcare service, i.e. provinces and districts, and healthcare providers. At this level, chief directors (and politicians too) see discretionary allocation of resources to be important for the strategic operation of healthcare services. The challenge is to ensure that HTA informs such allocation of funds so that all patients receive healthcare that is deemed safe and effective.

1. Herndon JH, Hwang R, Bozic KHJ. Healthcare technology and technology assessment. Eur Spine J 2007;16(8):1293-1302. https://doi.org/10.1007/s00586-007-0369-z 2. Croce D. Introduzione alla metodologia dell’HTA e Suo utilizzo, Tecnica Ospedaliere.

Technologie E Management Per La Sanità. Arro XLW-Numero 6-giugo, 2017.

3. Canadian Agency for Drugs and Technologies in Health. Guidelines for the Economic Evaluation of Health Technologies (3rd ed.). Ottawa: Canadian Agency for Drugs and Technologies in Health, 2006. https://www.cadth.ca/media/pdf/186_ EconomicGuidelines_e.pdf (accessed 9 November 2017).

4. Goodman CS. HTA 101: Introduction to Health Technology Assessment. 2014. https://www.nlm.nih.gov/nichsr/hta101/HTA_101_FINAL_7-23-14.pdf (accessed 1 November 2017).

5. Shemer J, Shani M, Tamir O, Siebzehner MI. Health technology management in Israel: HTA in action. Int J Technol Assess Health Care 2009;25 (Suppl 1):s134-s139. https://doi. org/10.1017/s0266462309090540 6. European network for Health Technology Assessment. EUnetHTA Joint Action 2, Work Package 8. HTA Core Model version 3.0, 2016. http://www.corehta.info/BrowseModel. aspx (accessed 1 August 2017).

7. Lampe K, Mäkelä M, Garrido MV, et al. European network for Health Technology Assessment. The HTA Core Model – a novel method for producing and reporting health technology assessments. Int J Technol Assess Health Care 2009;25(Suppl 2):S9-S20. https://doi.org/10.1017/S0266462309990638 8. European network for Health Technology Assessment. EUnetHTA JA (2012 - 2015) Methodological Guidelines for Rapid Relative Effectiveness Assessment (REA). http:// eunethta.eu/outputs/new-application-hta-core-model-hta-core-model-rapid-relativeeffectiveness-assessment-pharma (accessed 14 November 2017).

9. Favaretti C, Cicchetti A, Guarrera G, Marchetti M, Ricciardi W. Health technology assessment

29

November 2017

in Italy. Int J Technol Assess Health Care 2009;25(Suppl 1):S127-S133. https://doi. org/10.1017/S0266462309090539 10. Lehoux P. The Problem of Health Technology: Policy Implications for Modern Health Care Systems. New York: Routledge, Taylor & Francis Group, 2006.

11. Department of Health Autonomous Government of Catalonia, Catalan Health Service.

EUnetHTA Work Package 8. EUnetHTA Handbook on Health Technology Assessment Capacity Building. Barcelona: Catalan Agency for Health Technology Assessment and Research. 2008. http://www.eunethta.eu/sites/default/files/sites/5026.fedimbo.belgium. be/files/EUnetHTA%20Handbook%20on%20HTA%20Capacity%20Building.pdf (accessed 3 June 2017).

12. Croce D, Tivey D, Mueller D. Adaptation of checklists and the EUnetHTA core model 3.0 for Sub-Saharan African settings. Workshop presented at Health Technology Assessment International XIV Annual Meeting, Rome, Italy, June 17 - 21, 2017.

13. Chamberlain A. Why do employees stay? A clear career path and good pay, for starters.

Harv Bus Rev, March 2016. https://hbr.org/2017/03/why-do-employees-stay-a-clearcareer-path-and-good-pay-for-starters (accessed 7 August 2017).

14. The National Board of Health, Danish Centre for Evaluation and Health Technology Assessment. Introduction of Mini-HTA. 2005. http://www.sst.dk/~/media/47C62A769EB C4E80A153F986C5348F55.ashx (accessed 1 November 2017).

15. Gagnon MP. Hospital-Based Health Technology Assessment: Developments to Date.

Pharmacoeconomics 2014;32: 819. https://doi.org/10.1007/s40273-014-0185-3 16. Kriza C, Hanass-Hancock J, Odame EA, et al. A systematic review of health technology assessment tools in sub-Saharan Africa: Methodological issues and implications. Health Res Policy Syst 2014;12:66. https://doi.org/10.1186/1478-4505-12-66 17. Kanavos P. Methods of Value Assessment of New Medical Technologies: Challenges and Limitations. Presentation at Advance-HTA dissemination workshop. 2015. http://www. advance-hta.eu/PDF/Santiago/Presentations_Sept15/3-Methods-of-Value-AssessmentChallenges-&-Limitations-LSE.pdf (accessed 1 November 2017).

18. Thokala P, Deunas A. multiple-criteria decision analysis for health technology assessment.

Value Health 2012;(15):1172-1181. https://doi.org/10.1016/j.jval.2012.06.015 19. Keeney RL, Raiffa H. Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Cambridge: Cambridge University Press, 1993. https://doi.org/10.1017/ cbo9781139174084

Accepted 29 August 2017.

This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0.

ARTICLE

Addressing healthcare-management capacity building: The story of the African Institute for Healthcare Management S M Sammut, MA, MBA, DBA candidate Health Care Management Department, Wharton School, University of Pennsylvania, USA; and Institute of Healthcare Management, Strathmore University, Nairobi, Kenya Corresponding author: S M Sammut (smsammut@wharton.upenn.edu)

The focus historically on human-resource needs in healthcare has been on the development and training of physicians, nurses and allied health professionals. While there is still a gap that remains in these necessary roles, another acute gap has emerged, that of a scarcity of professionally trained managers for health systems, health facilities and the other organisations that comprise the healthcare system.

Arguably, many of the issues in health equity, such as the migration of physicians and nurses across borders, the operational inefficiencies of facilities and dysfunctional supply chains derive from sub-optimal management. Throughout the emerging markets, particularly in Africa, there have been some attempts to address the management gap, but there have been few, if any, degree programmes established for healthcare management. A full-scale healthcare-management MBA programme has been established at Strathmore University in Nairobi, Kenya. While the creation and development of the programme proceeded effectively and on schedule, the principals discovered that filling the ranks of qualified and experienced faculty in healthcare management was challenging. The faculty roster relied heavily on the recruitment of professors from healthcare-management departments in the USA and the UK. The attempts to design courses with locally relevant materials also revealed a serious lack of solid research and teaching materials. This article describes an approach to addressing the scarcity of Africa-oriented healthcare-management faculty, as well as the need for managerial teaching and learning materials focused on the African health context.

Strengthen Health Syst 2017;2(2):30-33. DOI:10.7196/SHS.2017.v2.i2.56

As the United Nations’ Millennium Development Goals aspired towards, and the Sustainability Development Goals reinforce, healthcare is a human right, but it has been in scarce supply in many regions of the world, particularly in Africa. While it is the case that there is a pressing need for the human-resource development of physicians, nurses and other healthcare professionals on the continent, it is also true that the development of a professional class of healthcare managers is just as urgently needed at every level of the healthcare value chain, whether they be health-unit managers, chief executives of large hospitals, or civic leaders of health ministries.

While healthcare facilities are often managed by healthcare practitioners, the full range of necessary skill sets is neither taught as part of medical or nursing education, nor acquired through experience. By way of example, healthcare professionals who wish to transition into management, and undertake a Master’s programme, often find that the content misses key elements that support managerial practice and decision-making. The disciplines of economics and finance, human-resource management, operations research and supply-chain management, marketing management, relevant law, public-policy development, leadership

and change management, and other skill sets are badly needed in healthcare settings, especially in societies with a history of scarce resources. Indeed, Management Sciences for Health (MSH)[1] points out, competent leadership and effective management systems are critical components of any organisation facing complex challenges and pressure to produce sustainable results. Leadership and management are especially important to health-service organisations and their managers in this era of rapid change, health-sector reform, the HIV and AIDS epidemic, and the crisis in human resources for health.

The aim of any degree programme, whether offered by a business school, school of public health, school of public policy or other institution, must be to provide students with the requisite knowledge and skills to improve population and community health through the application of sound healthcare management principles.

Formation of the African Institute for Healthcare Management (AIHM) One of the greatest obstacles to creating and offering full-scale curricula in healthcare management at institutions of higher learning in Africa is the relative lack of experienced specialty

November 2017

30

ARTICLE faculty to teach the wide range of subjects (arguably more than 30 disciplines) that encompass healthcare management. In order to promote the development of faculty for this purpose, representatives of three institutions – each acting of their own accord – Strathmore Business School (SBS) at Strathmore University in Nairobi, Kenya, the University of Cape Town (UCT) Business School in South Africa (SA) and the School of Public Health of the University of the Witwatersrand (Wits) in Johannesburg, SA, together with a representative from the SA Ministry of Health, have formed the African Institute for Healthcare Management (the Institute, or AIHM) to serve as a consortium of academic institutions and individuals focused on building a pool of world-class faculty, trainers and researchers that can be shared among programmes in healthcare management throughout Africa, whether Master of Business Administration (MBA), Master of Health Administration, Master of Public Administration, executive or other programmes.

The central premise of the Institute is that in the immediate future, African universities will accelerate the offering of advanced degrees and executive programmes in healthcare management, through schools of business, public health, public policy, or other academic structures. Most universities will have the resources to hire a small number of faculty to address some of the topics relevant to healthcare management, but it is unlikely that any one institution will have the resources to identify, hire and retain the full range of faculty needed to address the dozens of different specialised topics that such degrees or training programmes require, especially if the preference is to engage faculty on a fulltime basis and encourage top-tier research.

Participation in AIHM The Institute is open to multiple institutions in different parts of Africa. At the time of its formation in 2013, the board concluded that the Institute must develop evidence that it could be effective before seeking financial support and recruiting a wide range of members.

The board’s attention, therefore, focused on the new healthcaremanagement MBA programme at the SBS at Strathmore University, a private institution in Nairobi, Kenya. Strathmore admitted its first two cohorts in 2013 (19 students) and 2014 (34 students), both graduating in June of 2016. Three additional cohorts, with nearly 100 students, are currently in the programme. The students are from throughout the counties (i.e. provinces or states) of Kenya, which are now transitioning to a devolved health system. For the first time, the fifth cohort includes two students from outside Kenya.

AIHM is raising donor and grant funds for the purposes of recruiting, hiring and managing a group of faculty – ‘Institute Fellows’ – who can be shared and exchanged among participating institutions. The vision is to form a cadre of 20 or more Fellows over a period of 3 - 5 years. Fellows will be a mix of people at different points in their careers, and selected to assure the greatest possible range of expertise in the disciplines of healthcare management.

While Fellows will typically be drawn from the academic world, experienced healthcare managers seeking to move their careers into teaching and research will also be eligible. For the younger Fellows, and those transitioning into academia, the Institute will provide postdoctoral fellowships at major affiliated institutions in

31

November 2017

the USA, the UK and other countries, in order for them to hone their teaching and research skills. The Institute will allow participating institutions to draw from its pool of fellows – on a rotational basis – to teach courses in their respective curricula.

The Institute will not offer courses directly to university students or executives – that is the province of the participating institutions.

The mission of the Institute is to promote the development and availability of experienced faculty for use by the participating institutions, as well as to foster research in healthcare management and economics, and develop teaching and learning materials.

The Institute does not require that participating institutions follow a specific curriculum. Rather, each participating institution develops its own degree programme(s), devises the related curriculum and applies all its own policies and procedures to the programme’s administration. The design of any given degree programme is typically driven by an assessment of regional and local needs in healthcare management, epidemiological assessments and the guidance of accrediting agencies – an advisory function that the Institute will provide – but the preponderance of issues common to African countries suggests that there will be significant congruence in courses and content across the participating institutions, even in different degree programmes.

The availability of a shared faculty that addresses the multitude of necessary subjects enhances the ability of any institution to provide comprehensive degree and executive-training programmes.

The Institute will maintain a library of curricula, course syllabi in use at participating institutions and teaching and learning materials as a reference resource. It will make recommendations, but it will not prescribe the use of any one programmatic approach.

The Institute is an inclusive organisation for all institutions in Africa committed to improving and expanding managerial human resources in healthcare through degree programmes, executive programmes or research. In addition, the Institute invites organisations, companies and individual scholars who share its ideals and objectives to participate as members. As such, the Institute has a variety of membership categories designed to meet the needs and organisational structures of interested institutions, non-academic organisations and individual professionals.

Leadership and governance AIHM at the current time has a board of directors comprised of six individuals, who serve as the founders. Although each has a professional affiliation, they currently represent themselves individually on the board. They are: Dr Shadrick Mazaza, Professor, Graduate School of Management, UCT (Chair) Dr Debashis Basu, Professor, School of Public Health, Wits Dr Thabo Lekalakala, head, North West Department of Health, South Africa Dr Ben Ngoye, former Director of the Strathmore Healthcare MBA programme Dr Felix Olale, partner, Healthcare Investments, Leapfrog

Investments, and former Academic Director of the Strathmore Healthcare MBA programme Stephen Sammut, Senior Fellow, Health Care Management

ARTICLE Department, Wharton School, University of Pennsylvania, and co-founder and visiting professor of the Strathmore Healthcare MBA programme.

The Institute will also have two advisory boards: Academic Advisory Board. The board of directors shall determine the structure and composition of the Academic Advisory Board for the purpose of providing ongoing academic direction for the Institute and its Fellows in course content and research.

Industry Advisory Board. The board of directors shall determine the structure and composition of the Industry Advisory Board for the purpose of providing ongoing commercial direction for the Institute and its Fellows in course content and research.

Case study: The Healthcare-Management MBA programme at SBS Strathmore University is a private institution in Nairobi. Among its schools is SBS, founded in 2006. The new programme in healthcare management at SBS was launched in August 2013. The curriculum provides an example of the range of courses that often comprise a programme in healthcare management and leadership. In the case of SBS, the programme is structured as an MBA, and includes the courses customary to a general MBA, but with those core courses structured around content relating the disciplines under study to healthcare. The SBS curriculum was designed around an assessment of needs ascertained from numerous healthcare leaders in Kenya, East Africa and other parts of the world. It also took into account assessments made by MSH and IntraHealth.

The Strathmore MBA in Healthcare Management parallels the MSH teaching by developing: • the management and leadership of priority health programmes, health organisations, and multisectoral partnerships • the management systems of health organisations in the public and private sectors • the governance and management of health organisations and multisectoral partnerships.

In addition, the SBS programme imparts the full range of knowledge necessary to guide the ethical operation of healthcare organisations, including accounting, economics, finance, human-resource management, quality management, supply-chain management, organisational behaviour, change management and medical sociology.

For illustrative purposes, the curriculum currently in place at SBS includes the following courses:

Core and leadership course titles • • • • •

Financial and Managerial Accounting in Healthcare Organisations Healthcare Entrepreneurship and New Venture Management Healthcare Management Information Systems Quantitative Analysis and Statistics for Healthcare Management Organisational Behaviour and Change Management in Healthcare Organisations • Strategic Management in Healthcare Organisations • Managerial Health Economics

• Research Methods in Healthcare Management • Marketing Management for Healthcare Enterprises • Management Communications and Media Relations in the Healthcare Environment • Decision Analysis for Healthcare Managers • Healthcare Organisation Ethics and Governance.

Advanced healthcare courses • • • •

National Public and Private Healthcare Systems Managing Healthcare Human Resources Managing for Quality Patient Care and Efficient Operations Financial Management of Healthcare Organisations (corporate finance applied to healthcare organisations) • Healthcare Law and Policy • Healthcare Financing and Health Equity (financing of a healthcare system, including insurance).

Pending electives • • • •

Epidemiology as Applied to Management Medical Anthropology and Sociology in Healthcare Management The Pharmaceutical, Biotechnology and Medical-Device Industries Healthcare Delivery for Traditional Cultures.

Students are also required to complete a dissertation on an approved healthcare topic.

The curriculum is presented here by way of illustration. It is not intended to delineate a programme to be adopted by other member institutions. It is available in whole or in part for other institutions.

The need for AIHM Using the scope of the SBS curriculum as an indicator, a typical graduate programme in healthcare management, in addition to requiring faculty qualified to teach traditional core courses built around healthcare content, would need up to 30 additional instructors to address the application of the core courses to healthcare management situations and the various topics under the headings of the course titles above. SBS is approaching this challenge with a combination of current faculty members who are committed to adding material relevant to healthcare, local guest faculty and faculty drawn from the Wharton School, Johns Hopkins University, the Harvard School of Public Health, Boston University, the University of North Carolina School of Public Health, the Anderson School of Management at the University of California, Los Angeles, and other institutions. While this approach has proven effective, and provides all the opportunities needed for learning by the students, it would be even more effective with a specialised, dedicated faculty that focuses on African healthcare issues. For this purpose, the visiting international faculty have prepared the local Strathmore faculty to assume full teaching responsibility for the majority of the courses in the curriculum.

A programme offered under the auspices of a school of public health or public policy would likely have significant congruence with the content of an MBA programme, but with obvious differences related to the core missions of those schools. For

November 2017

32

ARTICLE example, a school of public health might want to emphasise more epidemiology, community-health principles, health indicators and specific health services such as maternal and child health. A public-policy school, similarly, would likely place more emphasis on the legal grounding of healthcare services, and the research and development of new policies addressing health needs. Having pointed out these differences, however, any degree programme purporting to prepare students for a career in some aspect of healthcare management would likely provide learning opportunities similar to those in the SBS curriculum. Similarly, institutions offering healthcare-management education through a business curriculum have a lot to learn from schools of public health and public policy, to say nothing of the rich contribution that can be made to curricular content by associate members and practitioner members of the Institute.

A regiment of Fellows associated with an institute can provide teaching services across most of the spectrum of topics required in any particular degree programme. As mentioned above, researchers in healthcare economics or aspects of management tend to focus

33

November 2017

on local geographies. Developing a critical mass of faculty, i.e. a concentration beyond the means of any one institution, primarily focused on the specific needs of African countries and regions, is most likely to occur when that faculty is assembled in Africa.

Establishing a consortium of Fellows is a powerful, cost-effective and sustainable strategy to meet those needs and objectives.

Conclusion and an invitation As the Institute develops, deans and faculty of business or public-health schools throughout Africa are invited to reach out to the author or any one of the board members for further discussion. The Institute is eager to expand healthcaremanagement education throughout the continent, and to offer its services in curriculum design, course content and faculty development.

1. Management Sciences for Health. Leadership, Management and Governance. http:// www.msh.org/our-work/health-system/leadership-management-governance (accessed 15 October 2011).

This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0.

ARTICLE

A hospital-management training programme in South Africa S Naidoo,1 MMed; M Mothagae,2 MBA, MPH; B Kistnasamy,3 FCCH; C Jinabhai,4 FCCH; D Basu,5 FCPHM Department of Community Health, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa Charlotte Maxeke Medical Research Cluster (CMeRC,) Johannesburg, South Africa 3 South African National Department of Health, Pretoria, South Africa 4 Faculty of Health Sciences, University of Fort Hare, Eastern Cape, South Africa 5 Department of Public Health Medicine, Faculty of Health Sciences, University of Pretoria, South Africa 1 2

Corresponding author: S Naidoo (Shan.Naidoo@wits.ac.za)

Background. The South African (SA) National Department of Health (NDoH) has identified the training of managers of public hospitals as key to improving efficiency in health-service delivery. As a part of that process, the NDoH, together with the universities of KwaZulu-Natal (UKZN) and the Witwatersrand (Wits) and the government of France, has launched a Master’s programme to train hospital managers. Its development began in 2003, and was initially based on the French hospital-management training programme. The aim was to make it a prerequisite for appointment as a senior hospital manager in the near future, although this was not achieved. The programme also aims to realise the NDoH’s policies of revitalisation and decentralisation of hospital management, by empowering managers and equipping them to effectively address the challenges of providing equitable and efficient health-service delivery. The aims of the Master’s course in hospital management are five-fold: (i) to develop uniform standards for the training of hospital managers; (ii) to develop leadership and managerial capacity among prospective and current hospital managers; (iii) to apply these competencies to critical research, intervention, evaluation and policy-development efforts in hospital management; (iv) to train adequate numbers of hospital managers for SA; and (v) to develop hospital management as a recognised profession among health and other professionals in SA. The programme was launched in 2006, and 150 students (chief executive officers (CEOs) of public hospitals) were enrolled during the 4-year duration of the programme. They were selected by the Training Unit for Hospital Managers of South Africa (the collaboration unit created by the abovementioned partners) based on selection criteria that it developed. An evaluation was carried out to assess the programme.

Objectives. To do a baseline audit of the current skills and competency levels of students enrolled in the Master of Public Health (MPH) in hospital management programme, and to assess the functioning of the hospitals they have been managing.

Methods. A cross-sectional study design was used. The study participants were the specific cohort of students (n=47) from public hospitals in SA who were enrolled in 2006 and 2007 at Wits in the MPH in hospital management programme. A structured questionnaire was distributed to the participants, and 41 out of the 47 students (30 first-year and 11 second-year students) completed it.

Results. The majority of the participants in the programme had been working in the public-health sector for a long time, and had acquired substantial practical experience. All of them had professional qualifications, as well as some management training or qualifications. Most of them had attended numerous management-related short courses. They suggested that the programme should be offered in such a way as to reinforce and consolidate their existing knowledge. They proposed that the NDoH should explore the possibility of recognition of prior learning. The participants supported the idea of continuing the current structure of the programme, and recommended the incorporation of soft skills such as the development of leadership, emotional intelligence and conflict-management skills. The participants preferred classroom teaching, case studies and experiential learning, and least supported the distance-based teaching methodology.

Conclusion. The findings from this study assisted the two universities and the NDoH to refine the programme currently offered, and confirmed that its strategy to professionalise hospital-management training was on the right track.

Strengthen Health Syst 2017;2(2):34-39. DOI:10.7196/SHS.2017.v2.21.63

Management training is fundamental to developing human resources for health. A lack of managerial capacity has been blamed for most health systems’ inefficiencies.[1] Healthcare systems need strong leadership if they are to be sustainable and responsive to the health needs of the future.[2] According to the World Health Organization,[1] effective leadership and management in the health services are key to using the available resources effectively and achieving measurable results. Leadership and management

skills have a positive impact on strengthening health systems,[3] and therefore these systems need strong leadership if they are to be sustainable and responsive to the health needs of the future.[2] Good leadership and management facilitate change within health organisations, and achieve better health services through the efficient and responsive deployment of people and other resources.

However, health management has proved a deceptively difficult and imprecise domain to grasp and define,[4] and the development

November 2017

34

ARTICLE of the adequate management and leadership skills that are needed in order to strengthen health systems has been given insufficient attention.[3] Capacity building of the managers working in the SA publichealth sector remains one of the priorities of the National Department of Health (NDoH). It was included in their Ten Point Plan,[5] as well as the Human Resources for Health Plan[6] and the Policy on the Management of Hospitals.[7] A number of universities in SA offer Master’s and doctoral education that seeks to produce skilled public-health practitioners, appropriately trained for the SA and African context. The programmes are all concerned with developing the capacity to address the major health challenges facing the public-healthcare systems of developing countries, particularly in Africa. Designed to meet the health-services management and research capacity building needs of Africa, each of these programmes aims to understand, mitigate the effects of, or challenge inequity within its disciplinary framework.

Prior to 1994, public-health training in SA focused exclusively on the specialist training of doctors. With the transformation to democracy, it became evident that established public-healthspecialist training was not meeting the need to build the humanresource capacity to manage the hospitals that consume a significant amount of resources within both the public and private health sectors. As a result, the NDoH has identified the training of the managers of public hospitals as the key to improving efficiency in health-service delivery. As part of that process, the NDoH, together with the Universities of KwaZulu-Natal (UKZN) and the Witwatersrand (Wits) and the government of France, has launched a Master’s programme to train hospital managers, and aims to make it a prerequisite for the appointment of senior hospital managers in the near future. The aims of the Master of Public Health (MPH) in hospital management are five-fold: (i) to develop uniform standards for the training of hospital managers; (ii) to develop leadership and managerial capacity among prospective and current hospital managers; (iii) to apply these competencies to critical research, intervention, evaluation and policy-development efforts in hospital management; (iv) to train adequate numbers of hospital managers for SA; and (v) to develop hospital management as a recognised profession among health and other professionals in SA. It was proposed that the programme would adopt an interdisciplinary

case-study and problem-orientated approach, with an emphasis on human resources, finance, operations, strategy, biostatistics and computing, project management, communication skills and operational-research methods, as applied to developing-country settings. Students were expected to complete a required set of core modules that cover these disciplines, and a range of elective modules that would build flexibility into the programme. A hospitalbased research project was intended to consolidate students’ competencies, skills and knowledge. Students were expected to conduct projects at selected hospitals identified in consultation with various stakeholders and role players. A workshop was organised by the NDoH in Durban in November 2005 to solicit opinion from various stakeholders and to finalise the content of the curriculum.

The two universities (UKZN and Wits) that launched the programme were chosen in 2005. A memorandum of agreement (MOA)[1] was signed between the NDoH and the universities to deliver the programme with technical and financial support from the French government, within the framework of an international agreement signed between the governments of SA and France, under the auspices of the European Union. The purpose of the programme, as highlighted in the MOA, was to develop specific courses and teaching material directly targeting the needs of SA public-service hospital managers. It aimed to accompany the national policies of revitalisation and decentralisation of hospital management, by empowering managers and equipping them to effectively address the challenges of improving service delivery.

A phased approach was put forward to achieve the vision of this programme (Table 1).

The programme was launched in 2006, and 150 students (chief executive officers (CEOs) of public hospitals) were enrolled during the 4-year duration of the programme. They were selected by the Training Unit for Hospital Managers of South Africa (the collaboration unit created by the abovementioned partners) based on selection criteria that it developed. An evaluation was carried out to assess the programme. Any programme should be subject to monitoring and evaluation to assess its impact and whether it has a beneficial effect and is adding any value to the health sector. It was specifically important to determine the impact of this programme on both the trainees and the hospitals where they work. However, it would not be possible to ascertain that any hospital-management improvements that might be experienced in the future are the

Table 1. Phased approach for the hospital-management programme Phase 1

Phase 2 Phase 3

35

• Development of the coursework components for an internationally recognised postgraduate programme in hospital management at Wits at certificate, diploma and degree level • Introduction of at least two courses in the above programme • Establishment of links between the university and selected public hospitals suitable for students’ hospital-based attachments • Full engagement of the French Department of Health as a partner in securing necessary expertise and resources, and in monitoring progress towards the achievement of goals • Extension of links to include public hospitals throughout SA as potential sites for students’ field-based attachments • Establishment of all the coursework and practical hospital-based components for the programme • Development of distance-based modules of the course • Development of an African hospital managers’ workbook • Seeking a link to other leading academic institutions in Africa with similar programmes, to develop regional nodes of excellence and a workable inter-regional academic framework

November 2017

ARTICLE result of this intervention without comparing it with a baseline.

In view of this, a baseline audit was done to assess the skills and competency levels of the students enrolled in the MPH in hospitalmanagement training at UKZN and Wits.

Methods The study was conducted on the cohort of students from public hospitals in SA who were enrolled in 2006 and 2007 at the two universities in the Master of Public Health (MPH) in hospital management programme. Sixteen students were enrolled in 2006 and 36 in 2007 at Wits. At the time of this survey, there were 14 students in their second year (the 2006 cohort), and 33 students in the first year (the 2007 cohort).

The scope of the study included an audit of the skills and competency levels of the cohort (trainees), and a baseline evaluation of the functioning of their hospitals. This baseline report would be used at the end of the project to evaluate the impact of the training programme on the improvement of hospital management.

A longitudinal cohort study design was used. A cohort was chosen and observed for a period of 3 years. A study of the skills and competencies of the cohort prior to completing the programme was therefore undertaken, which at the end of study period would serve as a baseline to measure the success of the intervention. No intervention was to be made by the researchers. However, recommendations were made to the NDoH based on the findings of the study. The audit was initially expected to include all 47 students enrolled in 2006/2007.

An information sheet was developed to explain to the participants the purpose of the project. A self-administered questionnaire was then administered to collect information from those students who were willing to participate. The data were analysed using the NCSS statistical software package (NCSS, USA).

Results The results obtained from the self-administered questionnaire (on the competency baseline audit of trainees) are described below.

Composition of the study group A total of 41 students participated (30 first-year and 11 second-year students) out of the total of 47, which is a response rate of 87% (90% among first-year and 78% among second-year students).

General characteristics of the subjects The general characteristics of the subjects are described below (Table 2). One of the participants reported having a physical disability.

Previous education and training The participants were asked about their educational background (professional and management training). The results are described in Tables 3, 4 and 5.

The majority (93%) of the participants had professional training in healthcare (Table 3). Six of them did not have an undergraduate degree. Some participants had completed various postgraduate courses (such as in family medicine or emergency care, a diploma in

Table 2. Demographics of the participants (N=41) First year, Total, n (%) n (%) Number 41 (100) 30 (73) Age group (years) 30 - 39 5 (12.5) 4 (13.8) 40 - 49 19 (47.5) 13 (44.8) 50 - 59 16 (40) 12 (41.4) Gender Male 20 (48.5) 15 (50) Female 21 (51.5) 15 (50) Ethnicity African 34 (85) 25 (86.2) Asian 1 (2.5) 1 (3.5) Coloured 4 (10) 3 (10.3) White 1 (2.5) -

Second year, n (%) 11 (27)

1 (9.1) 6 (54.5)

5 (45.5) 6 (54.5) 9 (81.8) 1 (9.1) 1 (9.1)

Table 3. Previous professional education and training, n (%) Medicine Nursing (Master’s) Nursing (BCurr Hons) Nursing (BCurr) Nursing (Diploma) Pharmacy (BPharm) Radiography (BTech) Environmental Health (BTech) Other (BA) Other (Diploma education)

6 (14.6) 3 (7.3) 7 (17.1) 13 (31.7) 6 (14.6) 1 (2.4) 2 (4.9) 1 (2.4) 1 (2.4) 1 (2.4)

tropical medicine and health and a certificate in TB/HIV care).

Thirty-one participants (75.6%) reported that they had some management qualifications (Table 4). Some (7) had more than one qualification. Three had completed a postgraduate degree and one had started but had not completed a Master’s degree in public administration.

The majority (85%) of the participants reported that they had attended some short courses (Table 5). Fifteen reported that they had attended many such short courses.

The above information shows that the majority of the participants had attended various levels of programmes, from a Master’s degree in management to various short courses. It is interesting to note that they still wanted to attend the MPH programme, which implies that they might not be satisfied with the above programmes, which had failed to give them the necessary skills for their jobs. Therefore it is a challenge to this Master’s programme to offer those skills.

Another interesting fact is the number of short courses attended by these participants. A number of short courses are offered by the private sector, which has substantial cost implications. Therefore, it is a challenge to the NDoH and the universities as to how they address the hunger for knowledge and skills of the existing health managers to equip them to become good and effective hospital managers.

November 2017

36

ARTICLE Health facilities

Work experience

The participants came from six provinces in SA (Table 6). The majority of them are from Level-1 (61%) and Level-2 (24.4%) health facilities.

Most of the participants were CEOs (85.4%) (Table 7). The majority of them had long experience in the health sector, and the average length of stay in their current post was 4.4 years. This indicates that this cohort of hospital managers is committed to the public health sector.

Table 4. Previous management education and training, participants’ qualifications, n (%) Qualification Master’s degree Public management Public administration Business administration Postgraduate diploma Management Human-resource management Graduate degree BCurr (Management) Graduate diploma Health management Health-service management Human-resource management Hospital management Nursing management Health and social welfare Public management Business management Business administration Certificate Hospital management Health management Integrated health and wellness programme management Public management Management

Working responsibilities Thirty-three of the participants perform only management responsibilities, while others also perform clinical and other responsibilities. The majority of them work after hours.

1 (2.4) 1 (2.4) 1 (2.4)

Necessary skills

1 (2.4) 1 (2.4)

Most of the participants identified the following skills as necessary for their work: • technical skills • soft skills (leadership, communication, time management) • coaching and mentoring • computer skills.

1 (2.4) 2 (4.8) 4 (9.8) 2 (4.9) 1 (2.4) 1 (2.4) 6 (14.6) 1 (2.4) 1 (2.4)

Some of them had acquired these skills through on-the-job training, while others acquired them through short courses provided by private institutions.

Among the participants, 8 had experience of mentoring both as mentor and mentee, 15 as a mentor only and 5 as a mentee only. All the respondents suggested that mentorship should be an integral part of this programme.

4 (9.8) 6 (14.6)

Table 6. Health facilities managed by students (N=41)

31 (75.6) 1 (2.4) 1 (2.4)

Table 5. Previous management education and training, short courses, N Courses Finance (including Public Finance Management Act and asset management) Human resources (including change management, wellness, labour relations, dispute resolution, motivation, performance management and development system, and job evaluation Project management Mentoring and coaching Gauteng Provincial CEO’s Training Programme District health management Occupational health GMDP Public service management Batho Pele TQM

8

10 7 2 2 1 2 1 1 1 2

CEO = chief executive officer; GMDP = Government Management Development Programme; TQM = total quality management

37

November 2017

Total, n (%) Province Free State Gauteng Limpopo Northern Cape North West Western Cape Health facilities Level 3 Level 2 Level 1 CHC/ Clinic Specialised hospital

First year, n (%)

Second year, n (%)

6 (14.6) 11 (26.8) 12 (29.3) 4 (9.8) 7 (17.1) 1 (2.4)

6 (20.7) 9 (31) 7 (24.1) 3 (10.3) 4 (13.8) -

2 (18.2) 5 (45.5) 3 (27.3) 1 (9.1)

2 (4.9) 10 (24.4) 25 (61) 3 (9.8)

1 (3.4) 9 (31.0) 16 (55.2) 3 (10.3)

1 (9.1) 1 (9.1) 9 (81.8) -

Table 7. Work experience (N=41) Position, n (%) Chief executive officer Clinical director Medical superintendent Nursing service manager Hospital manager Years of experience, mean (SD)

Total 35 (85.4) 0 2 (4.9) 3 (7.3) 1 (2.4) 4.4 (3.4)

First year 25 (83.3) 1 (3.3) 3 (10) 1 (3.3) 4 (2.9)

Second year 10 (90.9) 1 (9.1) 5.6 (4.3)

ARTICLE Current training programme The participants were asked to rate various teaching methodologies used in this programme (Table 8).

The analysis showed that most of the participants rated classroom lectures, case studies, experiential learning and group work highly. The two universities should seriously consider these recommendations from the participants.

Contents of the existing programme The participants were asked to comment on the suitability of the modules recommended at the Durban stakeholders’ workshop organised by the NDoH in Durban in 2005.

Most participants agreed that all the modules listed in Table 9 are important and should continue to be covered in this programme.

The participants suggested that more time should be allocated to the financial management, human-resource management, operations management and strategic management modules. They believed that they would gain a better understanding of hospital management from completing these modules. They also hoped to utilise the skills and knowledge acquired through this programme in managing their own hospitals.

Mentorship The following areas were covered during the programme: Patient Information, Financial Information, Procurement Information, HR Information, Clinical Management, Nursing Management and Hospital Management. The following areas of need were Table 8. Students’ approval ratings of teaching methodologies used Classroom lecture Group work Case study Distance-based work Computer-based work Seminar and workshop Experiential learning

Mean (SD) 4.7 (0.7) 3.9 (1.1) 4.1 (0.9) 3.1 (1.4) 3.3 (1.4) 3.4 (1.1) 4.3 (1.0)

5 = most preferable; 4 = preferable; 3 = neutral; 2 = not preferable; 1 = least preferable SD = standard deviation

Table 9. List of modules Health measurement (e.g. epidemiology, biostatistics) Human-resource management Financial management Operations management and logistics Problem-solving and decision-making Hospital disaster preparedness Institutional, corporate and hospitals management (corporate governance) Health policy and legislation Project management Strategic planning Control of diseases Leadership Research methodology

identified as a result of the project: (i) the integration of various areas of the hospital; (ii) good co-ordination between clinical and management divisions; (iii) a multidisciplinary team approach; (iv) a health-information system; (v) evidence-based practice to improve efficiency; and (vii) the application of theory to practice. The main limitations of the project were a lack of similar projects that had been undertaken in a hospital setting, meaning that the project team had to use its own ideas for the development of the project, and the unavailability of a best-practice model within SA. As a result, it was decided to use mentors from another country.

Opinions about future training Thirty-three of the participants (81%) believed that there should be a separate programme for a Master’s in hospital management. Only 3 participants (7%) suggested a generic Master’s in public health.

Two participants suggested a Master’s in business administration specialising in health.

Thirty-six participants (88%) believed that hospital management should be recognised as a professional career track in SA. In view of the above findings, the researchers recommended to the NDoH that it should consider creating a specialised qualification, a Master’s in Hospital Management, and explore the possibility of registering the qualification with a professional body such as the Health Professionals Council of SA.

Discussion The study highlighted the following important issues: • The majority of the participants in the programme had been working in the public health sector for a long time, and had acquired substantial practical experience.

• All participants had a professional qualification as well as some management training or qualifications. Most of them had attended numerous short courses. Therefore the MPH in hospitalmanagement programme should be offered in such a way as to reinforce and consolidate their existing knowledge. The NDoH should explore the possibility of recognition of this prior learning.

• The participants supported the idea of continuing the current structure of the programme, and also recommended the incorporation of soft skills into the programme.

• The participants preferred classroom teaching, case studies and experiential learning, and least supported a distant-based teaching methodology.

• The courses selected for the curricula are the same as those covered by the current programme at Wits and UKZN.

Conclusion The findings from this study assisted the two universities and the NDoH to refine the programme currently offered, and confirmed that its strategy to professionalise hospital-management training was on the right track.

Acknowledgements. The students who voluntarily participated in the study.

Author contributions. All authors contributed equally.

Funding. The study was funded by the NDoH, SA. Conflicts of interest. None.

November 2017

38

ARTICLE 1. World Health Organization. Towards Better Leadership and Management in Health: Report on an International Consultation on Strengthening Leadership and Management in LowIncome Countries. Geneva: WHO, 2007. http://www.who.int/management/working_ paper_10_en_opt.pdf (accessed 31 January 2017).

2. Ireri S, Walshe K, Benson L, Mwanthi MA. A qualitative and quantitative study of medical leadership and management: Experiences, competencies, and development needs of doctor managers in the United Kingdom. J Manag Market Healthcare 2011;4(1),16-29. https://doi.org/10.1179/175330304X10Y.0000000004 3. Rowe LA, Brillant SB, Cleveland E, et al. Building capacity in health facility management: Guiding principles for skills transfer in Liberia. Hum Resour Health 2010;8:5. https://doi. org/10.1186/1478-4491-8-5

39

November 2017

4. Hunter DJ, Brown J. A review of health management research. Eur J Pub Health 2007;17(1):Suppl 1:S33-S37. https://doi.org/ 10.1093/eurpub/ckm061 5. National Department of Health, South Africa. Ten Point Plan of Action. Pretoria: NDoH, 2009. http://www.sanews.gov.za/south-africa/10-point-plan-improve-health-system-sa (accessed 31 November 2017).

6. National Department of Health, South Africa. Human Resources for Health in South Africa.

Pretoria: NDoH, 2011. https://www.gov.za/sites/www.gov.za/files/hrh_strategy_0.pdf (accessed 31 November 2017).

7. South Africa. National Health Act (Act 61 of 2003): Regulation 656: Policy on the Management of Public Hospitals. Government Gazette No. 34522, 2011.

This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0.

PUBLIC HEALTH NOTEBOOK

Parametric hypothesis tests for the difference between two population means L N Dzikiti, MSc; B V Girdler-Brown, FCPHM, FFPH, BCom Hons (Econ) School of Health Systems and Public Health, Faculty of Health Sciences, University of Pretoria, South Africa Corresponding author: B V Girdler-Brown (brendangirdlerbrown@gmail.com)

When one wishes to perform a statistical-hypothesis test, the first important step is to select the correct, most appropriate, test to perform.

This article aims, firstly, to outline the test-selection criteria when one wishes to obtain statistical evidence about the equality of population means. Thereafter, Stata statistical software (StataCorp, USA) commands will be given for the various tests.

If the population means of interest are numerical and have known probability-sampling distributions, then the standard recommended statistical-hypothesis test, with data from samples, is either the classic t-test, or a variant of it.

In this article, the first part concentrates on the selection of the appropriate test to perform when using sample data to determine whether two population means are likely to differ. The selection criteria/steps are illustrated by a diagram (Fig. 1).

The second part describes how to perform the tests using Stata. In that section, some Stata output is also presented, and the interpretation of the output is explained.

Strengthen Health Syst 2017;2(2):40-46. DOI:10.7196/SHS.2017.v2.21.60

When one wishes to perform a statistical-hypothesis test, the first important step is to select the correct, most appropriate, test to perform. This article does not attempt to explain the underlying theory of the statistical tests it describes. Readers are urged to acquaint themselves with the theory by reading, or dipping into, a good textbook on the subject. There are many good textbooks available, but the one we recommend is by Pagano and Gauvreau.[1] Their explanations are clear, up to date and easy to understand.

Furthermore, the focus in this article is on the various twosample t-tests and how to perform them using Stata (StataCorp, USA). Therefore, manual calculations will not be described. In addition, details about how to perform single-sample tests, analysis of variance (ANOVA) (for when one wishes to compare more than two independent-sample means) and non-parametric tests, as important as these topics are, will not be covered here.

The authors have assumed a prior basic understanding of the principles of hypothesis testing, including the following concepts: the difference between population parameters and sample statistics; sampling error; the null hypothesis and the null value; alpha and the p-value; beta and and the power of a test; the 95% confidence interval (CI) for the difference between two means; type I and type II errors; and single-tailed v. two-tailed tests.

Selecting the appropriate test Parametric or non-parametric tests?

Hypothesis tests of the difference between two population means are performed using data from two samples that are assumed to have been selected in a random or probabilistic way. The null hypothesis is of the form: H0: µ1 - µ2 = 0 Two population means may be compared using either parametric or non-parametric (‘distribution-free’) hypothesis tests. Parametric tests are performed when one knows the sampling probability distribution for the difference between the two means, and the assumptions for the parametric tests have been met. Non-parametric tests are used when the nature of this sampling distribution is not known and cannot be surmised, or when the assumptions for a valid performance of the parametric test have not all been met. Although non-parametric tests have fewer restrictions than parametric tests, one should be aware that they also have conditions for their appropriate performance and these should always be checked for before embarking on a non-parametric test.

Whereas parametric tests will result in a p-value, as well as a 95% CI, for the difference between the two means, the non-parametric tests will only produce a p-value. The null hypothesis for a non-

November 2017

40

PUBLIC HEALTH NOTEBOOK

parametric test may not be that the two population means are equal. In addition, in general (certainly not invariably, however), non-parametric tests are less powerful than parametric tests. This is because non-parametric tests generally make use of less of the information contained in the sample.

For all these reasons, when performing a test to establish whether there is likely to be a difference between two population means, one should usually use a parametric test in preference to a non-parametric option, if conditions for the parametric test have been met.

Independent samples or paired samples?

The parametric tests for comparing means from independent samples are the t-test and the Welch test. The paired t-test is a suitable parametric test for comparing means from paired samples. Measurements are said to be paired when they are taken on the same unit of study (e.g. the same person or the same facility, depending on the unit of analysis).

In some cases, where two groups of study participants are very closely matched, it may also be acceptable to treat the two groups as ‘paired’.

When pairing is present, the paired t-test is in general more powerful than treating the two sets of readings as independent samples and then performing a t-test or Welch test. Therefore, if pairing is present, rather perform a paired t-test.

Some examples of paired data sets would include the weights of a group of people who were measured before the participants started a diet and exercise programme, and then measured again after a suitable interval. Another example might be sets of anatomical measurements of distances between surface landmarks on the left- and right-hand sides of the body.

Conditions for the t-test and the Welch test For the independent samples situation, valid use of either of the parametric hypothesis tests mentioned (the t-test and the Welch test) requires that for both samples being compared, the data are drawn at random from a population of data that have a normal, or bell-shaped, frequency distribution. If this assumption is not satisfied, or if there is uncertainty as to whether or not it is satisfied, then distribution-free methods should be considered, unless the sample sizes are large.

In practice, if the frequency distributions of the two samples appear approximately bell-shaped (unimodal and not too skewed) then it is safe to consider using these two parametric tests. If sample sizes are small, however, say <30, then one may wish to perform, first, a hypothesis test, such as a Shapiro-Wilk test, to assess whether each sample is likely to have been drawn at random from a normally distributed population of data.

For large samples (>30) it is unecessary to perform tests of normality, as the t-test and the Welch test are robust against departures from normality when samples are large. As a result, with large samples, there is also usually no need to first transform the data (for example, using log transformations with positively skewed data).

41

November 2017

One has to be clear, however, that for skewed data, even if the samples are large, the mean may not be an appropriate measure of central tendency, so one must first satisfy oneself that comparison of the differences between two means is a useful exercise.

Should one use the t-test or the Welch test?

These two tests are used to compare means from two independent samples, as in the situation where, for example, the mean birth weight of babies born to smoking mothers is compared to the mean birth weight of babies born to non-smoking mothers. However, there is a specific independent-samples t-test assumption that must be met for the t-test, in that the variances of the two samples to be compared must be equal. A Welch test should be used if the variances are not equal.

With the Welch test, the variance of the difference between the two means, as well as the degrees of freedom, are calculated differently from in the t-test calculations. As a result, the 95% CI for the difference between the two means, as well as the p-value, will be different from those obtained using a t-test.

When the variances are unequal, then the (inappropriate) t-test and the (appropriate) Welch test will often give quite different results. The differences in the results become smaller, however, as the differences between the variances become smaller, if the sample sizes are equal and if the sample sizes become larger. The extent of this similarity (between the two test results), however, varies depending on the size of the differences in the sample variances.

Performance of the Welch test does not require that the population variances should be unequal. The Welch test may be performed whether or not the population variances are equal.

The t-test, however, requires that the two population variances may be assumed to be equal. How do we decide whether the variances are ‘equal’ or not? Many older statistical textbooks suggest that the F-test be conducted to assess the equality of variances.

However, more modern text books, such as Pagano and Gauvreau,[1] discourage the use of the F-test to assess whether or not the two population variances may be considered to be equal. The F-test may lack sufficient power to correctly point the analyst away from an inappropriate t-test in many cases.[1,2] This is a particular risk when sample sizes are on the small side (say <20). We encourage our students to use visual inspection of the sample variances, and, if in doubt, to perform the Welch test rather than the t-test.

If the sample sizes of the two samples being compared are equal, then the Welch and t-tests give almost identical results. When these two sample sizes are both large and equal (say >30), the degrees of freedom (DF) are somewhat different (Welch v. t-test), but are so large as to not make any difference in practice.

When the sample sizes are equal but small (say <30), the DF are almost the same, so that once again, it makes no difference which of the two tests is used.

PUBLIC HEALTH NOTEBOOK

Some authors go so far as to suggest that the Welch test be used, rather than the t-test, in all situations.[3] We recommend the following approach (also illustrated in Fig. 1): • If sample variances are the same or vary by a very small degree only, use the t-test • In all other cases, or if in doubt, use the Welch test.

Conditions for the paired t-test For the paired t-test, the only assumption that must be met is that a single data set made up of the differences between each of the two paired readings should have a bell-shaped frequency distribution (‘unimodal and not too skewed’). This assumption becomes less important for larger sample sizes owing to the robust nature of the t-test.

Hence one should first calculate the differences between the two readings for each study participant, and then examine the frequency distribution of this newly calculated set of differences. This may be done by inspection of a frequency histogram, or, especially for small samples (<20), by performing a Shapiro-Wilk test.

This is especially important for small (say <20 pairs) studies. If the frequency distribution is clearly skewed, or not unimodal, then one should rather consider performing a non-parametric test, or perhaps transforming the differences to a form that has a bellshaped curve (if this is a meaningful thing to do; it may not be).

equivalent of the paired t-test is the Wilcoxon signed-ranks test. Both these tests can be easily performed in Stata. The null hypotheses may differ from those for the parametric tests, however, so they may not be truly ‘analogous’. In addition, be sure to check that the assumptions for the non-parametric tests have been met before performing these tests. Just because these tests are ‘distribution free’ does not mean that they are assumption free.

Single sample t-tests As illustrated in Fig. 1, one can also perform single-sample hypothesis tests where one compares the population mean to a fixed known value such as a gold standard, benchmark or target. For example, the mean normal birth weight of a sample of babies may be compared to an expected standard as defined by the World Health Organization. In this case a single sample t-test could be used. The null hypothesis for such a test is of the form: H0: µ = standard (where ‘standard’ = the gold standard, benchmark or target).

Once again, if the sample is small (say <30) the data should be unimodal and not too skewed. If this is not the case (for small samples), then a non-parametric test such as a single-sample sign test should rather be considered. As the sign test is less powerful (and actually assumes that the standard measure is a median rather than a mean), you should try to rather ensure that you have a large enough sample size (>30, say) so that you may use the t-test.

Non-parametric analogues of the parametric tests The usual non-parametric equivalent of the t-test is the MannWhitney-Wilcoxon (MWW) rank sums test. The usual non-parametric

Single sample

Two samples

n samples (n>2)

Single-sample t-test

Two-sample t-test

ANOVA

Independent samples unpaired t-tests

Paired samples paired t-test

Variances equal: t-test

Variances unequal: Welch test

Fig. 1. A flow chart to assist with the selection of an appropriate parametric hypothesis test of population means.

A brief note on the z-test (also sometimes referred to as a ‘normal test’) The z-test is rarely used nowadays. It is performed in the same way that one performs the t-test, except that it makes use of the population variance rather than the sample variance in calculating the CI and the p-value. This would seem desirable.

However, it is very rare that one knows the population variance (and does not know the population mean). The t-test procedure overcomes this problem (that we do not know the population variance) by substituting the sample variances for the population variances. As the sample variances are likely to be inaccurate estimates of the population variance (since they are subject to sampling error), this may sometimes result in unduly low estimates of the sample variances, resulting in type I errors.

With the t-test or Welch test this risk is mitigated by calculating wider CIs from the sample variances, and higher p-values, than would have been the case for a z-test. Of course, if the sample sizes are large then the z-test and t-test results will be similar even if one performs the z-test by using the sample variances substituted for the unknown population variances. For this reason, in some older textbooks, it was sometimes stated that one might perform a z-test if the samples were both, say, >30.

This option was considered desirable since the z-test does not require the normality assumption of the t-test if sample sizes are >30. However, simulation studies have shown that, for sample sizes

November 2017

42

PUBLIC HEALTH NOTEBOOK

>30, the t-test is sufficiently robust to give valid answers even if the samples are not drawn from normally distributed populations.

It is, therefore, acceptable to perform the t-test when the population variances are not known, as is usually the case for large sample sizes, even though the normality assumption is violated. It is perhaps for these reasons (the relative obsolescence of the z test) that Stata does not offer a z-test option as part of its standard package.

Performing the analyses using Stata statistical software One-tailed or two-tailed tests?

Having decided which hypothesis test to use, the next consideration is to decide whether one wishes to perform a one-tailed or a twotailed test. This decision does not affect the Stata command that you will use. Stata will present all the results for single upper tail, single lower tail and two-tailed tests.

One perfoms two-tailed tests if one is testing the null hypothesis that there is no difference between the population means. Since one does not wish to prejudge the outcome of the analysis, one is usually expected to perform a two-tailed test.

One encounters single-tailed tests, for example, in qualitycontrol studies. Let us assume that a provincial health department set a performance standard of at least 90% immunisation coverage for the measles vaccine in children aged 1 year for 2016.

The researcher would only be concerned whether the coverage were to be less than the performance standard set. There may be no interest in whether the standard had been exceeded (that would be a good outcome). In such a case, a single-tailed test of the null hypothesis that the coverage is <90% would often be performed.

Data layout Data may be entered into Stata in either the wide or the long format.

For paired tests, the data need to be in the wide format. For tests with independent samples, the data may be in either the long or the wide format. Below we give an example of data that are in the long (Table

1 A) v. wide (Table 1 B) format for independent samples (females and males are the two independent groups being compared).

The code used in Table 1 A for ‘female’ is 1 = female, while 0 = male. Note that in this case we do not have a ‘participant_id’ for the wide format, as the female with bodymass 65.3 kg cannot be the same person as the male with bodymass 60.9 kg.

It is usual in Stata to have data entered in the long format, as we would normally like to have participant_id for each entry, with rows of data that pertain to a particular participant. If there are variables that need to be entered in the wide format (such as repeated measures) this may easily be accommodated, so that the resulting dataset may contain some variables in the long format and some in the wide format.

Table 3 gives an example of both independent and paired data in the same table. Since there are also independent data present in the table, with these paired data, there is a participant_id entry specific to each participant row.

Pre- and post-treatment systolic blood pressures are measured in the same individuals and are thus paired data, which are presented alongside each other (in the wide format) in this case.

Selecting a value for alpha It is usual in biostatistics to use a p-value of 0.05 as a cut point for deciding if a result is statistically significant. The p-value that we obtain from the hypothesis test that we perform is the probability of obtaining the observed results, or more extreme results, by chance or sampling error if there really is no difference between the population means. Using p=0.05 as our cut point means we would reject the null hypothresis if p≤0.05. If this is the case, we might state that the test results are ‘statistically significant’. If p>0.05, we would fail to reject the null hypothesis, concluding that the results are not statistically significant.

This special value of p, namely 0.05, that is used to decide whether or not our results are statistically significant is called α (alpha). Note that we should never ‘accept’ a null hypothesis and/or conclude that two parameters are equal.

Table 1. The same data entered in the long format (A), and re-entered in the wide format (B) A. The long format layout Participant_id 1 2 3 4 5 etc.

Bodymass 65.3 60.9 54.4 59.1 67.2 etc.

B. The wide format layout Female_bodymass 65.3 54.4 59.1

Female 1 0 1 1 0 etc.

Male_bodymass 60.9 67.2

Table 2. A hypothetical example of a Likert-type questionnaire item I am satisfied with the clinic opening times Strongly disagree 1

43

November 2017

2

3

4

5

Strongly agree

PUBLIC HEALTH NOTEBOOK

Table 3. Data with a mix of long and wide formats, with paired data in the wide format Participant_id 1 2 3 4 5 etc.

Bodymass 65.3 60.9 54.4 59.1 67.2 etc.

Female 1 0 1 1 0 etc.

The reason why we failed to reject the null hypothesis might indeed be because the null hypothesis is true. However, it may also be due to the fact that we have samples that are too small, or that our test is underpowered, or that our measurement methods have been imprecise, and so on. We are not entitled to assume that the reason for statistical non-significance is that the null hypothesis is true.

Now, with a cut point of α=0.05, there is a probability that 5% of all null hypotheses that are true will test positive (type I errors) just because of sampling error (due to the variable composition of the samples that we are using and for no other reason). In other words, it is likely that we are wrong 5% of the time when we reject a null hypothesis and claim that our results are ‘statistically significant’.

If we set α=0.05, and then perform 100 pairwise t-tests of 100 known false hypotheses, using independent samples, the type I error would be 0.05 per test performed. The probability of committing at least one type I error when performing so many tests will exceed our planned level of 0.05.

When a researcher decides to collect data for a large number of variables, and then aimlessly perform pairwise hypothesis tests on all of them, in the hopes of finding ‘something significant’, the likelihood of finding statistical significance as a result of type I errors is increased.

Most textbooks are agreed that this is especially problematic when the decision to perform these hypothesis tests is made after data have been collected, although many do not agree that this is problematic if the tests are specified before data are collected.

However, just because a person lists every possible hypothesis test possible in the protocol, before collecting the data, (‘just in case’?) this does not diminish the risk of type I errors. Hence it is our view that in all cases where multiple pairwise hypothesis testing is carried out (either routinely or without good a priori arguments for their salience), a lower value of α should be required in order to establish statistical significance.

The Bonferroni adjusted value of α is a widely used adjustment (there are others) and is easily calculated as 0.05/T, where T is the proposed number of comparisons to be made. For five pairwise comparisons we would therefore use α=0.01 as the value to determine statistical significance (rather than 0.05).

A brief note on the analysis of Likert-style questionnaire data sets Likert-style questionnaires are of the following type where respondents are asked to check a single cell that best indicates their

pretreatmentsystolicbp 90 95 88 98 90 etc.

posttretamentsystolicbp 88 85 80 100 90 etc.

answer, as illustrated by Table 2.

One may then have two sets of responses, for example, one from a group of respondents working in the formal sector, and one from those who also work, but in the informal sector, on this issue.

One wonders if there is a statistically significant difference between the responses of these two groups. The values 1 - 5 are somewhat arbitrary, and are definitely neither continuous nor quantitative, although they are ordinal. Theoretically, one should not perform a t-test on these results since the data are qualitative.

The ‘amount’ of satisfaction represented by a move from 2 to 3 may not be the same as that between, say, 3 and 4 (if a ‘satisfaction’ amount could be quantitatively measured, which it cannot). Given that the data are ordinal at best, it would also be mathematically incorrect to calculate means or to perform addition, subtraction, multiplication or division on the data.

In addition, the responses to these Likert-type items are frequently skewed and bunched at one or other end. They may also be bimodal. Typically, they are not normally distributed.

Hence it would appear that a t-test or Welch test would be inappropriate on a number of counts. However, if one were to perform a non-parametric test, one would expect to lose power and run an increased risk of a type II error.

Alternatively, one may count the number of responses in each cell for each of the two comparison groups and then perform a χ2 test.

Unfortunately, if this approach is taken, one loses the information available from the ordinality of the responses; the χ2 test will treat the cells as if they were purely nominal counts, with no ordinal information being taken into account. Once again, power will be lost.

De Winter and Dodou[4] have evaluated the use of the t-test and also the non-parametric Mann-Whitney-Wilcoxon rank sum test in the situation where one has data from a five-option Likert item such as the example shown in Table 2. They performed this evaluation through empirical study (simulations) rather than theoretical argument. They conclude that, as long as one has samples of at least 10 respondents in each group, either the t-test or the Mann-WhitneyWilcoxon test may be used, in spite of the theoretical reservations that one might have regarding the use of the t-test in this situation.

They showed that the two tests had similar power even if the sample sizes of the two comparison groups were markedly different. When the data frequency distributions were skewed or peaked (as is commonly the case), the Mann-Whitney-Wilcoxon test had greater power than the t-test.

November 2017

44

PUBLIC HEALTH NOTEBOOK

We would, therefore, recommend that the Mann-Whitney-Wilcoxon test be used to analyse data from these Likert-style questionnaires with five ordinal selection categories. Not only is the use of this test theoretically easier to justify, but in most practical cases it would be expected to have greater power.

Stata commands (shown here between < and >; when typing the command omit < and >) 1. For H0: data were drawn at random from a population with normal distribution.

Preferred if a sample size <20. Reject H0 if p≤0.05. ( The Shapiro-Wilk test) (e.g. weight of females v. weight of males): <swilk weight if sex==0> <swilk weight if sex==1> 2. For H0: data were drawn at random from a population with normal distribution.

Preferred if a sample size >20. Perform visual inspection. <histogram weight if sex==0> <histogram weight if sex==1> 3. For obtaining the variances to decide by inspection if they are equal or not: If using Stata 14 or earlier: <sum weight if sex==0> Then, in order to obtain the variance (= standard deviation (SD)2): <di sd^2> (obtain SD from the output of previous command and insert). <sum weight if sex==1> Then, in order to to obtain the variance (=SD2): <di sd^2> (obtain SD from the output of previous command and insert).

4. For obtaining the variances to decide by inspection if they are equal or not: If using Stata 15 or later (more convenient way to obtain the variances): <ci variances weight if sex==0> This gives the variance directly. <ci variances weight if sex==1> This gives the variance

directly.

5. For a t-test: independent samples, data in wide format (less usual): <ttest weightmale = weightfemale, unpaired> Must put ‘unpaired’ 6. For a t-test: independent samples, data in long format (more usual): <ttest weight, by(sex)> No need to put “unpaired” 7. For a Welch test: independent samples, data in wide format (less usual): Stata calls the Welch test a t-test with unequal variances. <ttest weightmale = weightfemale , unpaired unequal> Must include ‘unpaired’.

8. For a Welch test: independent samples, data in long format (more usual): <ttest weight, by(sex) unequal> No need to include ‘unpaired’.

9. For a paired t-test:

45

November 2017

onsider a paired t-test for prediet and postdiet weights (‘prewt’ C and ‘postwt’): <gen diff = postwt-prewt> This generates a new variable called ‘diff’ that contains all the individual differences between pre- and postdiet weights). <swilk diff> This is the preferred way to assess if the differences are from a normally distributed population when sample size <20 pairs of data. <histogram diff> For samples >20 pairs a frequency histogram is produced and may be inspected. <ttest postwt = prewt> This will then result in the paired

t-test being performed. No need to type ‘paired’ in as in this wide format the paired test is the default in Stata.

10. For a single sample t-test: <ttest variable = GS> ‘Variable’ is the name of the single-sample variable. ‘GS’ is the gold standard/benchmark/target. Remember that these tests may often be single-tailed tests, especially in the context of quality control.

11. For a Mann-Whitney-Wilcoxon test: <ranksum weight, by(sex)> No need to type in either ‘unequal’ or ‘unpaired’. Data must be in the long format.

12. For a Wilcoxon signed-rank test: <signrank weightmale = weightfemale > Data must be in the wide format.

Stata version 15 outputs Example 1: Output from a t-test (independent samples, population variances assumed equal)

The first example, shown in Fig. 2, shows the output from a t-test (independent samples, population variances assumed equal).

Note that the two-tailed p-value is given by ‘Pr(|T|) = 0.2400’ (not statistically significant since p>0.05). The point difference between the two sample means is 1.221124, and the 95% CI for the difference between the means is given by –0.8379923 - 3.280239, which includes the null value of zero. This is expected since p>0.05 and the result is not statistically significant.

(The p-values given for Ha: diff <0 and Ha: diff >0 are for singletailed tests and need not concern us here).

Example 2: Output from a Welch test

The difference between the variances of the two samples (see Fig. 3) is very bizarre in this contrived example (6.432 and 0.742 or 41.34 v. 0.55). The resulting p-value (0.1381) for the two-tailed test is much lower than the p-value that would have been obtained had a t-test been performed (0.2306).

In spite of the fact that the degrees of freedom for the Welch test are lower than those for the t-test (12 v. 19), this does not mean that the Welch test is necessarily less powerful. The way in which the variance of the difference between the two means is calculated for the Welch test means that sometimes this variance may be smaller for the Welch test than it would have been for the t-test. As a result, one

PUBLIC HEALTH NOTEBOOK

cannot generalise about a difference in power (t-tests v. Welch tests), but should rather use the more appropriate of the two tests.

Note that the two-tailed p-value is given by ‘Pr(|T|) = 0.1381’ (not statistically significant). The point difference between the two sample means is 2.855769, and the 95% CI for the difference between the means is given by –1.0546 - 6.766139, which includes the null value of zero. This inclusion of the null value is expected since p>0.05.

In addition, note the variances SD2 are very different and the sample sizes are both different and small (13 males and 8 females).

The (inappropriate in this case) t-test would have yielded a p-value of 0.2306.

(The p-values given for Ha: diff <0 and Ha: diff >0 are for single tail tests and need not concern us here).

Fig. 2. Results of a t-test with independent samples (Stata output). Mean Body Mass Index for males (Group 0) v. females (Group 1).

Example 3: Output from a paired t-test

In the example presented in Fig. 4, the difference between two weights measured in gold miners, 1 year apart, was generated.

There were 510 participants (hence two samples of 510 readings, and a single sample of 510 differences). The Shapiro-Wilk test yielded a p-value of <0.001, indicating that the data were not drawn at random from a normal distribution. However, the frequency histogram yielded a unimodal graph that was not skew, nor particularly kurtotic (peaked).

This shows the importance of using the histogram, rather than the very sensitive Shapiro-Wilk test, to decide whether or not to proceed with the parametric t-test. In any event, with such a large sample size, the argument is academic; The t-test would have been an appropriate test to use in any case.

Note that the two-tailed p-value is given by ‘Pr(|T|) = 0.0441’. This result is, of course, statistically significant (p<0.05). The difference between the two sample means is 0.65, and the 95% CI for the difference between the means is 0.018888 - 1.281112 (which excludes the null value of zero). This is expected for a statistically significant result.

(The p-values given for Ha: mean (diff ) <0 and Ha: mean (diff ) >0 are for single-tailed tests).

Presenting and interpreting the results As a default, one might consider presenting one’s results correct to two decimal places, with p-values correct to three decimal places. Stata p-values of, say, p=0.0000 should rather be presented as p<0.001, since, theoretically, p cannot be zero.

The minimum information that should be presented includes the name of the test performed and the point estimate for the difference between the means, along with the p-value and the 95% CI for the difference between the two means. In the case of multiple tests having been performed, if a Bonferroni adjusted p-value is presented then this should be stated. Alternatively, if the p-value is unadjusted then the Bonferroni-adjusted α value should be stated alongside the results.

Fig. 3. Results of a Welch test (Stata output). Mean age for males (Group 0) v. females (Group 1) if BMI >25.

Paired t test Variable

Obs

Mean

Std. Err.

Std. Dev.

[95% Conf. Interval]

hb2 hb1

20 20

15 14.35

.3769685 .3101358

1.685854 1.386969

14.211 13.70088

15.789 14.99912

diff

20

.65

.3015312

1.348488

.018888

1.281112

mean(diff) = mean(hb2 - hb1) Ho: mean(diff) = 0 Ha: mean(diff) < 0 Pr(T < t) = 0.9779

t = degrees of freedom =

Ha: mean(diff) != 0 Pr(|T| > |t|) = 0.0441

2.1557 19

Ha: mean(diff) > 0 Pr(T > t) = 0.0221

Fig. 4. Results of a paired t-test (Stata output). Haemoglobin levels in 20 athletes before and after taking a naturopathic product for 4 weeks (fictitious data).

1. Pagano M, Gauvreau K. Principles of Biostatistics, 2nd ed. Pacific Grove: Duxbury, 2000.

2. Moser BK, Stevens GR. Homogeneity of variance in the two-sample means test. Am Stat 1992;46(1):19-21.

3. Delacre M, Lakens D, Leys C. Why psychologists should by default use Welch’s t-test instead of Student’s t-tests. Rev Int Psychol Soc 2017;30(1):92-101.

4. De Winter JCF, Dodou D. Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon. Practic Assess Res Eval 2012;15(11):1-16. http://pareonline.net/getvn.asp?v=15&n=11 (accessed 20 August 2017).

Accepted 31 August 2017.

November 2017

46