Volume XI - Numer II by Laboratório de Tecnologia e Sistemas de Informação

JISTEM JOURNAL OF INFORMATION SYSTEMS AND TECHNOLOGY MANAGEMENT REVISTA DE GESTÃO DA TECNOLOGIA E SISTEMAS DE INFORMAÇÃO

www.jistem.fea.usp.br

ISSN: 1807-1775

Volume 11 : Number 2 : 2014

Available Online Disponível Online Support:

JISTEM Journal of Information Systems and Technology Management Revista da Gestão da Tecnologia e Sistemas de Informação ISSN online: 1807–1775

Every four months/Quadrimestral

Universidade de São Paulo – FEA USP Prof. Dr. Marco Antonio Zago – USP Reitor/Rector Prof. Dr. Vahan Agopyan – USP Vice-Reitor/Vice-Rector Prof. Dr. Reinaldo Guerreiro - Diretor da FEA/Dean of FEA Editor Prof Edson Luiz Riccio, PhD. University of São Paulo – FEA, São Paulo Brazil Assistant Editor Marici Gramacho Sakata, PhD. TECSI University of São Paulo – FEA, São Paulo Brazil Editorial Board – Comitê de Política Editorial Armando Malheiro da Silva, University of Porto, Porto, Portugal Christophe Benavent, Université Paris Ouest Nanterre La Defense, Paris, France Henrique Freitas, Federal University of Rio Grande do Sul, Rio Grande do Sul, Brazil JaeJon Kim, Chonnam National University, Gwangju, Korea Luc Marie Quoniam, University Paris 8, Paris, France Michael D. Myers, University of Auckland, Auckland, New Zealand Miklos Vasarhelyi, Rutgers Business School, New Jersey, USA Rejane Maria da Costa, University of Brasilia, DF, Brazil Robert D. Galliers, Bentley College, Massachusetts, USA Editorial Review Board – Comitê Científico Editorial Adam Mazurkiewicz, Instytut Technologii Eksploatacji, Poland Adalberto A. Fischmann, University of São Paulo, São Paulo, Brazil Antonio Carlos dos Santos, Federal University of Sao Carlos, São Carlos, Brazil Birger Hjorland, Royal School of Lis, Copenhagen, Denmark Burak Arzova, Marmara University, Istanbul, Turquia Dennis F. Galletta, University of Pittsburgh, Pittsburgh, USA Emerson Maccari, Uninove, Sao Paulo, Brazil Fabio Frezatti, University of São Paulo, São Paulo, Brazil Fernando Colmenero Ferreira, University of Madeira, Madeira, Portugal Geraldo Lino de Campos, University of Sao Paulo, Sao Paulo, Brazil Gilson Schwartz, University of Sao Paulo, Sao Paulo, Brazil Guilherme Ari Plonski, University of São Paulo, São Paulo, Brazil Jan Capek, Univerzita Pardubice, Pardubice, Czech Republic Jose Dutra de Oliveira Neto, University of Sao Paulo, Sao Paulo, Brazil José Rodrigues Filho, Universidade Federal da Paraíba, Paraíba, Brazil Miguel Juan Bacic, University of Campinas, Campinas, Brazil Napoleão Verardi Galegale, Centro Paula Souza and Galegale Associados, Sao Paulo, Brazil Rosana Grillo Gonçalves, University of Sao Paulo, São Paulo Brazil Salvador Ruiz-de-Chavez, APCAM, Ciudad de Mexico, Mexico Published by TECSI - Laboratório de Tecnologia e Sistemas de Informação - Revista de Gestão da Tecnologia e Sistemas de Informação - EAC FEA USP Av. Prof. Luciano Gualberto, 908 FEA 3, Cidade Universitária - São Paulo/SP 05508-900 Brasil Fone: 55-11-3091 5820 r.190 Fax: 55-11-3091 5820 jistem@usp.br Indexation/Directories SciELO, Latindex, Proquest, Ulrich's Periodical Directory, DOAJ, The Index of Information Systems Journals, ACPHIS, Dialnet, Ebsco, Gale Infotrac, CLASE, Portal de Periódicos USP, Qualis CAPES, Cabell's Directory Webmaster jistem@usp.br Technical Support Equipe TECSI pesquisatecsi@usp.br Terms and Conditions The license lets others distribute, remix, tweak, and build upon your work, even commercially, as long as they credit you for the original creation. This is the most accommodating of licenses offered. Recommended for maximum dissemination and use of licensed materials. Direitos e Permissão Os artigos são de total responsabilidade dos autores e todos os direitos reservados ao TECSI. Esta licença permite que outros distribuam remixem e construam sobre a sua obra, mesmo comercialmente, desde que lhe deem crédito pela criação original.

Support:

JISTEM Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No.2, May/Aug. 2014, pp. 207-513 ISSN online: 1807-1775

Volume 11: Number 2 / Volume 11: Número 2

2014

Content / Índice

207-208

Editorial

209-210

IDENTIFYING OPERATIONAL REQUIREMENTS TO SELECT SUITABLE DECISION MODELS FOR A PUBLIC SECTOR EPROCUREMENT DECISION SUPPORT SYSTEM Mohamed Adil, Information School, University of Sheffield, Sheffield, UK Miguel Baptista Nunes, Information School, University of Sheffield, Sheffield, UK Guo Chao Peng, Information School, University of Sheffield, Sheffield, UK

211-228

THE USEFULNESS OF USER TESTING METHODS IN IDENTIFYING PROBLEMS ON UNIVERSITY WEBSITES Layla Hasan, Department of Computer Information Systems, Zarqa University, Jordan

229-256

A FUZZY MULTICRITERIA APPROACH FOR IT GOVERNANCE EVALUATION Angel Cobo, University of Cantabria, Santander, Spain Adolfo Alberto Vanti, University of Vale do Rio dos Sinos, São Leopoldo, RS, Brazil Rocío Rocha, University of Cantabria, Santander, Spain

257-276

USING ANALYTIC NETWORK FOR SELECTION OF ENTERPRISE RESOURCE PLANNING (ERP) ALIGNED TO BUSINESS STRATEGY UTILIZAÇÃO DA REDE ANALÍTICA PARA A SELEÇÃO DE SISTEMAS INTEGRADOS DE GESTÃO (ERP) ALINHADOS À ESTRATÉGIA DE NEGÓCIO Alberto de Medeiros Jr., Mackenzie Presbyterian University, São Paulo, SP, Brazil Gilberto Perez, Mackenzie Presbyterian University, São Paulo, SP, Brazil Sérgio Lex, Mackenzie Presbyterian University, São Paulo, SP, Brazil

277-296

297-326

THE EFFECT OF THE INTERNET ON THE PATIENT-DOCTOR RELATIONSHIP IN A HOSPITAL IN THE CITY OF SÃO PAULO Jayr Figueiredo de Oliveira, EAESP-FGV – Getúlio Vargas Foundation, São Paulo, SP, Brazil

327-344

STATE OF THE ART: INTEGRATING SERVICES FOR MEGA EVENTS Jorge R. B. Garay, University of São Paulo, São Paulo, SP, Brazil, Gustavo M. Calixto, University of São Paulo, São Paulo, SP, Brazil,

345-360

JISTEM, Brazil Vol.11, No.2, Mai/Aug. 2014, pp. 207-208

www.jistem.fea.usp.br

208 Content / Indice

Alexandre M. De Oliveira University of São Paulo, São Paulo, SP, Brazil, Marcelo K. Zuffo, University of São Paulo, São Paulo, SP, Brazil, 8

USING THE BSC FOR STRATEGIC PLANNING OF IT (INFORMATION TECHNOLOGY) IN BRAZILIAN ORGANIZATIONS Adriano Olímpio Tonelli, Instituto Federal de Educação, Ciência e Tecnologia de Minas Gerais, Formiga, MG, Brazil Paulo Henrique de Souza Bermejo, Federal University of Lavras, Lavras/MG, Brazil André Luiz Zambalde, Federal University of Lavras, Lavras/MG, Brazil

361-378

CREDIT ANALYSIS USING DATA MINING: APPLICATION IN THE CASE OF A CREDIT UNION ANÁLISE DE CRÉDITO POR MEIO DE MINERAÇÃO DE DADOS: APLICAÇÃO EM COOPERATIVA DE CRÉDITO Marcos de Moraes Sousa, Instituto Federal Goiano Campus Ceres, GO, Brazil Reginaldo Santana Figueiredo, Federal University of Goiás, GO, Brazil

379-396

PERFORMANCE MEASUREMENT OF INFORMATION TECHNOLOGY GOVERNANCE IN BRAZILIAN FINANCIAL INSTITUTIONS Sara C. Boni Barbosa, University of Sao Paulo, USP/FEARP, Ribeirão Preto, SP, Brazil Ildeberto Aparecido Rodello, USP/FEARP, Ribeirão Preto, SP, Brazil Silvia Inês Dallavalle de Pádua, USP/FEARP, Ribeirão Preto, SP, Brazil

397-414

AUTOMATED TEXT CLUSTERING OF NEWSPAPER AND SCIENTIFIC TEXTS IN BRAZILIAN PORTUGUESE: ANALYSIS AND COMPARISON OF METHODS Alexandre Ribeiro Afonso, University of Brasília/UnB, Brasília - DF, Brazil Cláudio Gottschalg Duque, University of Brasília/UnB, Brasília - DF, Brazil

415-436

AUDITORIA CONTÍNUA DE DADOS COMO INSTRUMENTO DE AUTOMAÇÃO DO CONTROLE EMPRESARIAL Washington Lopes da Silva, TecBan S/A, São Paulo, SP, Brazil Jorge Rady de Almeida Jr., University of São Paulo, São Paulo, SP, Brazil

437-460

RESULTADOS DO 11º CONTECSI – CONGRESSO INTERNACIONAL DE GESTÃO DA TECNOLOGIA E SISTEMAS DE INFORMAÇÃO OUTCOMES OF THE 11th CONTECSI – INTERNATIONAL CONFERENCE ON INFORMATION SYSTEMS AND TECHNOLOGY MANAGEMENT Edson Luiz Riccio, University of São Paulo, São Paulo, SP, Brazil Marici Cristine G. Sakata, TECSI FEA USP, University of São Paulo, São Paulo, SP, Brazil Nelma Terezinha Zubek Valente, Universidade Estadual de Ponta Grossa, Paraná, Brasil Ligia Capobianco, TECSI University of São Paulo, São Paulo, SP, Brazil

461-510

Events / Eventos

511

Contributions / Submissão de Artigos

JISTEM, Brazil Vol.11, No.2, Mai/Aug. 2014, pp. 207-208

512-513

www.jistem.fea.usp.br

JISTEM Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No.2, Mai/Aug. 2014, pp.209-210 ISSN online: 1807-1775

DOI: 10.4301/S1807-17752014000200000

Editorial Edson Luiz Riccio, Editor Dear Readers It is well known that Journals play a significant role in the dissemination of knowledge as well as they support important discoveries. Although JISTEM does not publish special issues in selected research streams, the Editorial board, Editor and reviewers are always sensitive to determining when submissions are being innovative. That makes each edition a rich and relevant resource for academics as well as practitioners. If the research results of the published articles reach the right target, it is hard to know. But as Straub and Ang (2011) states: “The way assuredly not to examine this question (Is information systems research relevant to practice?) is by asking practitioners whether they read scholarly journals. They do not. Nor should they. Scholarly journals are written by scholars for scholars. In many cases, the scientific apparatus is so complicated and sophisticated that only a subset of scholars can fully appreciate specialist articles. But if scientific articles have merit for praxis then this knowledge can and should be shared. It is conceivable that the essential principles and implications of academic research can be communicated to practitioners by the best IS academics. In this second issue, May – August of 2014, we are pleased to present a selection of 11 highly relevant articles plus the Report: Outcomes of the 11th CONTECSI USP. The articles cover the following main topics:             

Decision Models User Testing Methods Fuzzy Multicriteria Approach Analytic Network for Selection of Integrated Management Systems (ERP) Information Technology Governance Internet on the Patient-Doctor Relationship Integrating Services BSC for Strategic Planning Credit Analysis Using Data Mining Performance Measurement of IT Governance Automated Text Clustering Continuous Auditing Outcomes of the 11th CONTECSI USP– International Conference on Information Systems And Technology Management – May 28 to 30, 2014

JISTEM maintains the high quality and visibility in both national and international sectors. In addition to authors from UK, Jordan and Spain, we also present Brazilian authors from Rio Grande do Sul, Minas Gerais, Sao Paulo, Goias and the Federal District. We thank all authors and reviewers and wish you a pleasant reading.

* Straub, D. and Ang, S. (2001) MIS Quarterly Vol. 35 No. 1 pp. iii-xi/March

R. Gest. Tecn. Sist. Inf. /JISTEM Journal of Information Systems and Technology Management, Brazil

210 Content / Indice

Editorial Edson Luiz Riccio, Editor Caros Leitores, Já é bem conhecido o fato de que as Revistas Científicas desempenham um papel significativo na disseminação do conhecimento, assim como apoiam importantes descobertas. Embora a JISTEM não publique edições especiais em temas de pesquisa selecionados, o Comitê Editorial, Editor e revisores estão sempre atentos a identificarem artigos inovadores. Esse fato transforma cada edição em uma fonte rica e relevante para os acadêmicos e profissionais. Se os resultados das pesquisas dos artigos publicados alcançam o alvo desejado, isso já é algo mais difícil de saber. Contudo, de acordo com Straub and Ang (2011): “A maneira que certamente não devemos usar para examinar essa questão (As pesquisas sobre sistemas de informação são relevantes para a prática?) é perguntar aos profissionais se eles leem revistas científicas. Eles não as leem e nem devem. As revistas científicas são escritas por acadêmicos para acadêmicos. Em muitos casos, o aparato científico é tão complicado e sofisticado que apenas um subgrupo de acadêmicos consegue apreciar plenamente os artigos especializados. Porém, se os artigos científicos têm seu mérito para a realização da prática, o conhecimento então pode ser compartilhado. É concebível o fato de que os princípios essenciais e as implicações das pesquisas acadêmicas possam ser comunicados aos profissionais, pelos melhores acadêmicos em sistemas de informação. Nesta segunda edição de Maio a Agosto de 2014, temos o prazer de apresentar uma seleção de 11 artigos altamente relevantes e também o Relatório: Resultados do 11º CONTECSI USP. Os artigos cobrem os seguintes tópicos principais:             

Decision Models User Testing Methods Fuzzy Multicriteria Approach Analytic Network for Selection of Integrated Management Systems (ERP) Information Technology Governance Internet on the Patient-Doctor Relationship Integrating Services BSC for Strategic Planning Credit Analysis Using Data Mining Performance Measurement of IT Governance Automated Text Clustering Continuous Auditing Resultados do 11o CONTECSI USP – Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação – de 28 a 30 de maio de 2004.

A JISTEM mantém a alta qualidade e visibilidade, nos setores nacional e internacional. Além de autores do Reino Unidos, Jordânia e Espanha, apresentamos também autores brasileiros do Rio Grande do Sul, Minas Gerais, São Paulo, Goiás e Distrito Federal. Agradecemos a todos os autores e revisores e desejamos a todos uma agradável leitura.

* Straub, D. and Ang, S. (2001) MIS Quarterly Vol. 35 No. 1 pp. iii-xi/March

Vol. 11, No.2, Mai/Aug. 2014, pp.209-210

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 211-228 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200001

IDENTIFYING OPERATIONAL REQUIREMENTS TO SELECT SUITABLE DECISION MODELS FOR A PUBLIC SECTOR EPROCUREMENT DECISION SUPPORT SYSTEM Mohamed Adil Miguel Baptista Nunes Guo Chao Peng Information School, University of Sheffield, Sheffield, UK ___________________________________________________________________________________________

ABSTRACT Public sector procurement should be a transparent and fair process. Strict legal requirements are enforced on public sector procurement to make it a standardised process. To make fair decisions on selecting suppliers, a practical method which adheres to legal requirements is important. The research that is the base for this paper aimed at identifying a suitable Multi-Criteria Decision Analysis (MCDA) method for the specific legal and functional needs of the Maldivian Public Sector. To identify such operational requirements, a set of focus group interviews were conducted in the Maldives with public officials responsible for procurement decision making. Based on the operational requirements identified through focus groups, criteria-based evaluation is done on published MCDA methods to identify the suitable methods for e-procurement decision making. This paper describes the identification of the operational requirements and the results of the evaluation to select suitable decision models for the Maldivian context. Keywords: Multi-Criteria Decision Analysis (MCDA); Decision Model; Focus Group; Public Sector; e-Procurement

1. INTRODUCTION Procurement is challenging, because making a fair judgment in selecting the right suppliers, according to a large number of sometimes subjective criteria, is never easy. The stakeholder theory by Freeman (1984) gives no room for negligence in public sector procurement because of the impact of the decision on a vast number of stakeholders. Information and Communication Technology (ICT) provides important assistance to the decision making process in procurement by enabling the processing ___________________________________________________________________________________________ Manuscript first received/Recebido em: 28/11/2013 Manuscript accepted/Aprovado em: 26/06/2014 Address for correspondence / Endereço para correspondência Mohamed Adil, Information School, University of Sheffield, Room 224, Regent Court, 211 Portobello Street, Sheffield S1 4DP, UK, E-mail: m.adil@sheffield.ac.uk Miguel Baptista Nunes, University of Sheffield, United Kingdom, E-mail: j.m.nunes@sheffield.ac.uk Guo Chao Peng, University of Sheffield, United Kingdom, E-mail: g.c.peng@sheffield.ac.uk Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

212

Adil, M., Nunes, M. B., Peng, G. C.

of large amounts of data and information from various suppliers. This faster and more accurate processing provides a better platform to select the best supplier, i.e. decision makers produce sounder judgements. In the stage of supplier selection, decision making is based on the performances of the suppliers against a pre-set list of criteria. These criteria allow public sector institutions to state differentiated priorities when they announce for bids or tenders. The DSS uses Multi-Criteria Decision Analysis (MCDA) to evaluate procurement alternatives, due to the need to conciliate a large number of evaluation criteria. The study uses published mathematical models to study and select suitable decision models for a public sector e-procurement DSS. The basis for this paper is derived from a research project that aims to develop a decision model for an e-procurement Decision Support System (DSS) for the public sector in the Maldives, especially focusing on the education sector. For the main research project, at first a literature review of public sector procurement and MCDA methods was done. Secondly, a field research involving a set of focus groups was conducted with public education sector officials to identify operation requirements for a suitable evaluation method. Thirdly, a criteria-based evaluation was done on MCDA methods to identify the most suitable methods based on the identified operational requirements. Finally, a series of quantitative analysis was conducted on the resulting methods from criteria-based evaluation to finalise an MCDA method. This paper aims to present the identification of the operational requirements with key results of the field research. The paper also presents the final results of the criteria-based evaluation to select suitable decision models for the Maldivian context and illustrated rejection of a decision model and a selection of a decision model based on the evaluation. 1.1

BACKGROUND

The objectives of public sector procurement are similar as in the private sector (Leenders & Fearon, 1997). The objective of procurement in both cases is to purchase the right quality of material, at the right time, in the right quantity, from the right source, at the right price (Baily, Farmer, Jessop, & Jones, 1994; England, 1967; Leenders & Fearon, 1997; Weele, 2000). In order to achieve this macro-objective, there are four major principles in public procurement. They are non-discrimination, equality, transparency and proportionality (Weele, 2010). The features and characteristics of public sector procurement are based on these principles, which led to the development of strict guidelines that rule procurement processes in the public sector today. These guidelines were adopted as general guidelines for specific countries or work categories in others, but are usually developed and implemented locally through best practice (Rowlinson & McDermott, 1999). Nonetheless, due to its visibility and inherent transparency, public sector procurement has adopted a rigid structure and it is difficult to make changes in public purchasing as it is established by law and strict regulations (Leenders & Fearon, 1997). Public procurement must adhere to the guidelines provided by public authorities. In every country, public procurement must comply with the specific legislative requirements (Falagario, Sciancalepore, Costantino, & Pietroforte, 2012). The guidelines provide systematic approaches for tender evaluation. The public purchasing laws require that the contract be awarded to the lowest capable bidder who JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

213

sector e-procurement decision support system

fits into the requirements laid by the invitation for bid (Brown, Wright, Cloke, Morris, & Trumper, 1984; Falagario et al., 2012; Leenders & Fearon, 1997). Public procurement has limited flexibility and narrow evaluation criteria when dealing with bid evaluation due to its legal bindings (Leenders & Fearon, 1997). In Maldives, Dhaulathuge Maaliyyathu Gaanoonu 2006 (literally, Public Finance Act 2006) and Dhaulathuge Maaliyyathuge Gavaaidhu 2009 (literally, Public Finance Regulation 2009) are the governing laws and regulations for the public sector procurement (Dhaulathuge Maaliyyathu Gaanoonu 2006). The responsible body for governing the public sector finance is assigned by the President of Maldives (Dhaulathuge Maaliyyathu Gaanoonu) and it is the Ministry of Finance and Treasury (MoFT) (Dhaulathuge Maaliyyathuge Gavaaidhu 2009). Therefore, the MoFT creates and amends Public Finance Regulation requirements (Dhaulathuge Maaliyyathu Gaanoonu). In alignment with (Brown et al., 1984), in the Maldives the method of procurement in the public sector also varies depending on the product, value, urgency, location, suppliers capability (Dhaulathuge Maaliyyathuge Gavaaidhu). In normal circumstances the method of procurement mainly depends on the value of procurement product as described by Leenders & Fearon (1997) and also based on public finance regulations of the Maldives (Dhaulathuge Maaliyyathuge Gavaaidhu). Based on Dhaulathuge Maaliyyathuge Gavaaidhu, mode of procurement is categorised into three different bands based on the cost of procurement as follows:  If procurement material or service cost is less than MVR1,000.00 (approximately GBP42.00), the institution can purchase it at common market rate from any supplier.  If the procurement material or service cost is from MVR1.000.00 to MVR25,000.00 (approximately GBP1.042.00), the institution should get quotations from three different suppliers and purchase it from the best supplier.  If the procurement material or service cost is from MVR25.000.00 to MVR1.500,000.00 (approximately GBP62.500.00), the institution has to go for public bidding and form the Bid Evaluation Committee (BEC) to evaluate the bids.  If the procurement material or service cost is MVR1.500,000.00 or above, still the institution has to announce a public tender and the tenders are evaluated through the National Tender Board (NTB). According to these Maldivian regulations, if the procurement material or service cost is MVR25.000.00 or above, public sector institutions have to announce public bidding or tendering. Tendering and bidding are differentiated through their value in the research context as follows:  If a procurement costs between MVR 25.000.00 to MVR1.500,000.00, it is regarded as a bid and evaluated by BEC.  If a procurement costs MVR1.500,000.00 or more, it is regarded as a tender and evaluated by the NTB. However, a tendering process and a bidding process are the same but the members represented by evaluation of the suppliers are different (Dhaulathuge Maaliyyathuge Gavaaidhu). Members are of the BECs are the senior officials of the JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

214

Adil, M., Nunes, M. B., Peng, G. C.

institute for which procurement takes place. The NTB is monitored by the MoFT and it represents members from different sectors. The research focuses on the band where procurement material or service cost is between MVR25.000.00 to MVR1.500,000.00 which requires public bidding and evaluated by the BEC in public education sector. Selected education institutions are allocated with autonomous budget and with the approval of the Ministry of Education (MoE) these institutions do their own procurement. Budgets of other institutions are maintained by the MoE based on the needs of the institutions. For these institutions the MoE does the tendering and the BEC of the MoE selects the suppliers. These tenders are sometimes group tenders for multiple institutions to make the procurement less costly. For procurement the institutions announce request for tender (RFT). In the announcement it specifies a date for information session before submission date. During the information session the evaluation criteria are provided to potential bidders. The BEC uses pre-announced criteria and corresponding weights of the criteria set for every procurement for bid evaluation. Every bid has more than one criterion to be used in each evaluation. Use of multiple criteria creates a situation where the application of MCDA methods becomes appropriate for bid evaluation. Although MCDA and multi-criteria decision making (MCDM) are used interchangeably, â&#x20AC;&#x153;in a decision making context, MCDA would imply some sort of standard by which one particular choice or course of action could be judged to be more desirable than another. Consideration of different choices or courses of action becomes an MCDMâ&#x20AC;? (Belton & Stewart, 2002). Multi-objective decision making (MODM) and multi-attribute decision making (MADM) also present the same class of methods (Triantaphyllou, 2000). MCDA is one of the most well-known branches of decision making (Triantaphyllou, 2000) and particularly useful when a wide number of factors are to be considered in the decision (Amponsah, 2011). Furthermore, even though MCDA has a wide variety of methods, they all have certain common features. These features are the concept of alternatives and attributes (Triantaphyllou, 2000). In simple terms, the available choices and the evaluation criteria are fundamental for any MCDA method. There are numerous MCDA methods used in different disciplines including procurement. This paper aims to identify a set of suitable ones for the Maldivian context. 2. METHODOLOGY This research addressed two major different aspects that determine the suitability of an MCDA method to the public education sector context: the operational requirements, which are heavily influenced by the above mentioned legal framework, and the specific characteristics of particular MCDA methods. These two aspects are evaluated against each other to identify suitable decision models for the e-procurement DSS, as shown in research design in Figure 1.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

215

sector e-procurement decision support system

MCDA Methods

Operational requirements Criteria-based evaluation

DSS Model

Figure 1: Research Design

In the first phase, a critical literature review identified all of the most common MDCA methods which were distilled into the 41 potential candidates out of more than 80 found, as presented in Appendix 1. The aim of this study is to identify the characteristics of the particular MCDA models in order to evaluate it against public sector requirements. In the second phase, a field research was conducted involving a set of focus group interviews and the analysis of official documentation which include procurement related regulations and selected documents of evaluated procurements with results. The aim of the field research is to identify the operational requirements for public sector procurement in research context. Finally, a third phase involved a criteria-based evaluation of the MCDA methods performed against the operational requirements of the Maldivian context gathered through the field research. The aim of this evaluation analysis is to check which of the identified MCDA methods comply with public sector requirements and could become good candidates for the design and development of a new eProcurement Decision Support System. 3. IDENTIGYING OPERATIONAL REQUIREMENTS A field research was carried out to identify operational requirements. This field research involved focus group discussions with BEC members of public sector education institutions. Focus groups are being increasingly used in academia and the method is being followed in a variety of social sciences (Barbour & Kitzinger, 1999). Focus groups are used to gather information on public perceptions and viewpoints (Lloyd-Evans, 2006). The main strength of this data collection method was described by Morgan (1997) by stating that two eight-person focus groups would generate as many ideas as 10 individual interviews, but the conduction of the data collection process (2 sessions instead of 10), transcription and analysis are clearly more efficient and less time consuming. â&#x20AC;&#x153;Focus groups are group discussions exploring a specific set of issues. ... Crucially focus groups are distinguished from the broader category of group interviews JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

216

Adil, M., Nunes, M. B., Peng, G. C.

by the explicit use of group interaction to generate data. Instead of asking questions of each person in turn, focus group researchers encourage participants to talk one another” (Barbour and Kitzinger 1999:4) Focus groups are best applied to look for people’s experiences, opinions, wishes and concerns (Barbour & Kitzinger, 1999). Group discussions match the ways in which participants’ views are generated, spoken, and exchanged in everyday life. These discussions also help validating statements and views through corrections by the group concerning views that are not correct (Flick, 2009). Since the focus groups in this research project aimed to understand and identify operational requirements (i.e. to identify how the current process works and what the expected characteristics are and the ideal evaluation method), it was expected that the group members would contribute to refine and add missed points offered by colleagues. That is, it was expected that the focus group would stimulate the respondents, help them recall meaningful events and provide a complete and socially negotiated view of operational requirements. In this sense, the information obtained would clearly surpass what could have been acquired from answers in single interviews (Flick, 2009). 3.1 Sample Size and Sampling Strategy Barbour and Kitzinger (1999) state that the number of focus group interviews to be undertaken in a specific study could vary from three or four interventions to more than fifty, that is “the appropriate number of focus groups will depend on the research question, the range of people you wish to include and, of course, time and resource limitations” (Barbour and Kitzinger, 1999:7). Moreover, Barbour and Kitzinger (1999) believe that statistical representativeness is not intended for the most focus group research projects. Consequently, most focus groups research uses structured sampling rather than random sampling, depending on the research question (Barbour & Kitzinger, 1999; Morgan, 1997). In the case of this research, the Maldivian MoE represents 213 schools for which most of the procurements are handled by the MoE’s procurement department. However, depending on the procurement type, the MoE may delegate the procurement process to 19 different schools. There are 20 distinct BECs for 213 schools and the MoE. The participants in the focus group discussions for this research context involved the senior officials of the MoE and other government institutions represented as the BEC. Since the participants are senior officials and every BEC has limited number of representatives, it was decided to involve those decision makers that are engaged in the largest number of procurement processes and minimise the number of focus groups. During the initial meetings with the MoE, it was stated that every institution under the MoE followed the same procedures and had similar expectations, mimicking the procurement processes at the MoE level. However, the researchers have requested to have focus groups at least with three BECs in three different locations with three different institutions to verify if there are any differences and to enrich the data. Thus, the focus groups were conducted with the BECs of the MoE itself, a primary school and a secondary school which cover the major spectrum of procurements under the JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

217

sector e-procurement decision support system

MoE. More specifically, these three focus groups would include BEC members who make procurement decisions for 196 schools out of 213 schools and the MoE, covering BECs for 92 percent of the public sector institution under the MoE. 3.2 Group Size and Composition Since the main advantage of focus group is to have group interaction, careful consideration of focus group composition, such as shared interest in group members, is crucial in the facilitation process and in the final results obtained (Lloyd-Evans, 2006). Consequently, focus group researchers tend to agree that involving people with similar shared experiences is more productive. Barbour & Kitzinger (1999) go even further by stating that many researchers prefer to work with pre-existing groups who are already working together. Therefore, this study did not mix BEC members from one institution with those from the other institution in the focus groups. Since the BEC has limited members and the members are senior staff as described previously, having multiple responsibilities other than the BEC function, it was not realistic to expect that every single member of the BEC would be available for the focus group. Therefore, the size was set to a minimum of three members and a maximum of all BEC members of the target focus group. 3.3 Recruitment and Research Setting As there are no defined rules for the recruitment of focus groups (Lloyd-Evans, 2006), the recruitment was done initially having a meeting with a director of the MoE followed by formal meetings arranged with the heads of the chosen institutions to arrange focus group meetings with the BECs of the institutions. The institutions allocated meeting times based on the availability of at least a minimum requested size of BEC members for the focus group. As suggested by Barbour & Kitzinger (1999), the sessions were held in meeting rooms of the institutions where the focus group was conducted making it easier for the participants to attend. Additionally, these rooms were ideal as they are quiet, comfortable and free from interruptions. 3.4 Recording and Transcribing Traditionally, focus group recording ranges from the most basic level of note taking to audio and video recording. Some researchers recommend video recording because it can provide additional gestural and interaction information. However, the interpretation of these later type of communication may be difficult to interpret, to handle and may provide the misleading illusion of comprehensiveness (Barbour & Kitzinger, 1999). Focus group transcription can be difficult because of sudden leaps, and interruptions by members (Barbour & Kitzinger, 1999). To try and minimise this problem, this research used video recording which helped to easily identify individual speakers and their gestures. It also helped to accurately keep the sequence of the discussion in transcription. This enabled the researchers to overcome the aforementioned difficulties in transcription. The video recording also helped to recap JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

218

Adil, M., Nunes, M. B., Peng, G. C.

some of the very technical discussions through the verbal and non-verbal explanations that occurred during the discussion. 3.5 Analysing and Presenting Focus Group Data Analysing focus group data usually follows similar processes to any other qualitative data analysis. However, the group context requires analysis of group contribution as a whole rather than just the individuals’ contribution. However, it is particularly difficult to keep the balance between concepts provided by the group and individual ideas within it (Barbour & Kitzinger, 1999). The analysis involves comparing discussion of similar themes and examining how these relate to the variation between groups (Barbour & Kitzinger, 1999; Lloyd-Evans, 2006) and between individuals (Barbour & Kitzinger, 1999). Moreover, sometimes respondents may change their positions later in the discussion which transforms the analysis of focus group in quasi “detective work” (Krueger, 1998). This research used thematic analysis as proposed by King and Horrocks (2010). All focus groups were conducted in Dhivehi (local Maldivian language) and transcribed in the same language. However, all the coding was done in English. The transcripts were coded in a three-tier approach, starting with descriptive coding, followed by interpretive coding and finally defining overarching themes. The transcripts included gestural information (e.g. “left member nods in approval” or “centre member looks surprised and doubtful”). This information was kept as part of the quotations in order to illustrate the discussion, but the object of coding was the verbal information. For the purpose of formulating the narrative that forms the proposed inductive theory, important and supporting quotations were translated into English. 4. KEY RESULTS OF THE FIELD RESEARCH It was evident, as the MoE had mentioned in the preliminary meetings, that all the institutions follow the same procedure for procurement. It repeatedly reinforced at multiple times during the focus groups that the tender evaluation method had to be inline with the Maldivian legal requirements, mainly to safeguard the BEC, the institution, suppliers and the public as a whole. It was repeatedly stated that the BEC is a legal body to evaluate bids consisting of senior public officials. BECs are required to explicitly express the reasons for selection or rejection of any bid. For the decision making context, bid evaluated results are the reason for selection or rejection. “Based on this evaluation, the last part of the sheet (evaluation sheet) should state that ‘this’ is the best supplier and all the attendees of the committee need to sign it (bid evaluation sheet)” (F2ML2). Public Finance Regulation states two compulsory bid evaluation criteria, namely price and duration of delivery, and this was confirmed by the participants in the focus group: "Generally price is a criterion […] Depending on the procurement there may be other allocated points (weights)" (F2ML2).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

219

sector e-procurement decision support system

"Points are awarded for duration [of the delivery], experience [of the supplier] and capability of supplier as well" (F2ML3). In addition, other criteria were mentioned: 

Financial capability of the supplier.



Technical capability of the supplier.

 Justifiability of the prices submitted by suppliers compared to the estimated price of the procurement material or service.  execution. 

Similar past experiences of the suppliers in terms of size and its Any other important criteria perceived by the public sector department.

Nonetheless, the most common strategy on selection is to give the major focus on price by setting higher weight for this criterion whilst other criteria are also being considered. However, in specific cases, such procurement of technical equipment, more importance may be given to the required specification rather than the price. “For instance, supplier whose delivery is the fastest may not meet our spec. But the highest cost bid may exactly meet our spec. In such cases, we go for the one which meets the spec” (F2ML1). Before the bidding process starts, the public sector institution will set formal dates for application as well as announce the dates for an “information session for the bidders” (sometimes also referred to as “pre-bid meeting”). This meeting will provide explanation and discussion of the written details of requirements of the procurement products or services. The meeting also explicitly discusses the evaluation methods and the criteria and weights used, which are also part of the formal documentation. “Pre-bid meeting is compulsory for procurement of more than MVR25.000. The information sheet provided in the meeting will have the specification for the procurement and marking criteria, allocated points for price, allocated points for delivery, allocated points for experience and so on” (F2ML2). Since currently BECs are using the weighted sum for all public procurements, it is expected that the potential bidders know how the calculations are done. However, in some information sessions, the evaluation calculations are explained. This information session provides the opportunity for the bidders to clarify their doubts. If the answers are readily available, it is provided during information the session, otherwise sent to all the bidders after the session, either by email or by telephone. “We explain [how the calculations are done] very rarely” (F2ML3). “The people who attend bid meetings commonly will know [the calculations] [… but] there may be questions from bidders. Those queries will be answered during the session. If the queries are not answered during the meeting, we provide answers later either by email or phone” (F2ML2). Performances for all the criteria of each individual successful bidder are evaluated by the BEC. No bid can be discarded a priori based on an unexpected low level of performance of any of the criteria. In explaining the process, a BEC member provided the following illustration:

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

220

Adil, M., Nunes, M. B., Peng, G. C.

“For instance, we know the market value of a copier is MVR25.000. Even if a bidder submits a bid with MVR200.000 for the same copier, we need to evaluate the whole bid. However, there can be disqualified bids, may be due to missing required documents, which will not be brought for evaluation” (F2ML2). However, BECs are also concerned about unreasonable lower bid prices by the bidders to win the bid. In such cases, the BEC estimates the cost and if any bid is below a reasonable cost estimate, the bid gets cancelled stating the reason. All BEC members who attended the evaluation session need to sign the evaluated sheets. Some suppliers get suspended for a certain period of time by the BEC due to previous undesirable experiences with them. In such cases, bids from those suppliers are not considered until the suspension is over. “[ … Suppliers] can withdraw their bid after bidding. However, after agreement, if they reject it, bid committee may suspend them for 6 months” (F2ML2). In evaluation, each criteria is individually evaluated in isolation, it is not assumed that the criteria influence each other and no change is made in point allocation based on such an assumption, that is, “each criterion is evaluated based on its own merits” (F2ML1). After evaluation, the BEC decides who the winner is and the winner is informed through a letter. No separate letters are sent to the bidders who did not win. However, the letter to the winner gets copied to all the bidders, so that every bidder knows who won the bid. The calculated evaluation results are not sent to bidders. However, if any bidder requests the calculations, it is then provided but hardly any bidder requests for the details. “Instead of sending an individual letter to unsuccessful bidders, we copy the awarding letter to all bidders […] If anyone wants to see [the evaluation], we show them this [evaluation] sheet” (F2ML2) In terms of MCDA method, BECs are currently using weighted sum method. No other methods have been tried in evaluation of bids in the Maldivian Public Sector. BECs assume that the weighted sum provides accurate results for evaluating the best bid. BEC members, not normally being statisticians, are not always aware that there are other more suitable methods for bid evaluation. This research aimed at breaking this tendency and at investigating if any other MCDA methods can be used. When asked, BEC members were somewhat evasive but expressed the belief that any other method to be adopted would have to be in line with law and regulation and would have to be approved by the Ministry of Finance in a first instance and ultimately by the Parliament. This research aims at forming the basis for such a proposal. In general operational terms, BECs believe that a suitable evaluation method should be: 

Clear, simple but accurate;



Time saving;



Comply with regulations;



“Avoid” (interpreted in this study as minimise) complaints;

 “No chance” (interpreted in this study as minimising the chance) of manipulation from “both sides” (i.e. suppliers and evaluators); JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

221

sector e-procurement decision support system



“Trustable”, error free.

Therefore, these are the operational criteria that need to be combined with legal criteria to enable the selection of a suitable MCDA method. A full list of operational requirements, emerged from the field research, are presented in Appendix 2. 5. ILLUSTRATION OF DECISION MODEL SELECTION In order to identify a suitable MCDA method that fits the Maldivian context, a criteria-based evaluation as defined by Chen et al (2012) as an “evaluation that is conducted according to predefined checklists, heuristics or principles” was done. In the case of this research, the criteria-based evaluation of the MCDA methods is performed against operational requirements of the Maldivian context gathered as described above. After a thorough analysis, two methods – TOPSIS (Technique for Order Preference by Similarity to an Ideal Solution) and COPRAS (COmplex PRoportional Assessment) – emerged as applicable for the Maldivian context and capable of meeting operational requirements. This process was reported in-depth in other publications. However and to illustrate the use of the operational and legal requirements identified in this paper, a rejected method and a selected method is presented as follows. 5.1 Multi-Attribute Utility Theory Multi-Attribute Utility Theory (MAUT) was rejected as not suitable for operational reasons. MAUT aims to obtain the maximum overall utility with tradeoffs of the attainment of some objectives against other objectives. The method develops a utility function based on the decision maker’s preference structure, and the utility function is used to find optimal solution (Sanayei, Mousavi, Abdi, & Mohaghar, 2008). Huang (2011) states that MAUT is a quantitative method which has an orderly process to identify and analyse multiple variables to find a solution. By applying the developed MAU (multi-attribute utility) function a decision maker can find the utility of every alternative, to identify the alternative with highest utility to select. The expression of MAU function given by Huang (2011:399) is as follows:

where n is the number of attribute, ui is a single-attribute utility function over attribute i, wi is the weight for attribute i and = 1 (0 ≤ wi ≤ 1 for all i ). Many different utility elicitation methods are developed to find decision maker’s MAU function, which can be a holistic approach such as a multiple regression analysis or a decomposed approach (Huang, 2011).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

222

Adil, M., Nunes, M. B., Peng, G. C.

Sensitivity analysis is a part of MAUT procedure (Min, 1994). If inconsistency is found in MAUT, the preference information of the decision maker has to be changed. (Moshkovich, Mechitov, & Olson, 2005). However in public sector procurement as mentioned above, the weights are announced in advance and even if the decision is not favourable based on sensitivity analysis, it is not possible to make changes to weights. MAUT needs decision makerâ&#x20AC;&#x2122;s involvement to develop the utility function. When it is done, it can be used to evaluate many alternatives. Decision makerâ&#x20AC;&#x2122;s efforts are no more needed even if a new alternative is to be considered. Sensitivity analysis could balance the likely inaccuracy in the measurements. However, there is no psychological justification for the questions faced by decision maker (Moshkovich et al., 2005). To use MAUT, a special training should be done by decision makers and MAUT does not consider likely human errors in evaluation (Moshkovich et al., 2005). Public sector procurement evaluations are done by senior officials of the public sector and they have responsibilities other than the BEC limiting their time. Undergoing training and holding onto it during MAUT analysis incorporate practical difficulties. In addition, the BEC may change from time to time and would require the training to be conducted every time a new member joins the BEC. Since MAUT has mismatches with public sector procurement requirements as discussed above, MAUT is rejected. 5.2 Technique for Order Preference by Similarity to an Ideal Solution Technique for order preference by similarity to an ideal solution (TOPSIS) is accepted as suitable for the Maldivian context based on the legal and operational requirements. TOPSIS is an MCDA method to rank alternatives from a finite set of alternatives. The basic principle is to minimize the distance to the ideal solution while maximizing the distance to the negative-ideal solution for the chosen alternative (Jahanshahloo, Lotfi, & Davoodi, 2009; Olson, 2004). TOPSIS method is criticised due to the issue of the satisfactory level for both criteria of the shortest distance from the ideal solution and the farthest distance from the negative ideal solution, because TOPSIS does not consider the relative importance of those distances (Opricovic & Tzeng, 2004). In addition to this, with Euclidean distance as used in TOPSIS, the closest alternative to the positive ideal solution is not necessarily the farthest alternative from the negative ideal solution (Aghajani & HadiVencheh, 2011; Chamodrakas, Leftheriotis, & Martakos, 2011). Bottani & Rizzi (2006) states that the major weakness in TOPSIS could be the need for monotonic criteria. However, Bottani & Rizzi (2006) believes that TOPSIS works well for one-tier decision trees, while AHP (Analytic Hierarchy Process) is preferable for widely spread hierarchies, and this is where AHP could become competitive against TOPSIS. A further drawback stated by Tsaur (2011) is that a narrow gap between the performed measures is derived in the normalized scale for each criterion due to the operation of normalized decision matrix. Therefore, with a narrow gap in the method, it is not good for ranking and cannot reflect the true dominance of alternatives. When drawbacks are compared to the public sector procurement as discussed in operational requirements, having monotonic criteria could be considered, as public sector requires pre-announced criteria with its weights for each bid or tender which JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

223

sector e-procurement decision support system

cannot be changed later in the procurement process. The next drawback of having narrow gap between the performance measures after normalisation still provides relative comparative figures, even though small, which can still be used for calculation. Finally, the public sector is looking for the best alternative based on the performance criteria as described in public sector requirements. The best alternative should be the alternative closest to the ideal solution even if the alternative is not the furthest from the negative ideal solution. Therefore, in the public sector procurement context, the best alternative by TOPSIS, by chance not being the furthest from the negative ideal solution is acceptable as the alternative that will be the closest to the ideal solution. Considering the above mentioned drawbacks in TOPSIS have no direct conflict with public sector requirements, the method is considered appropriate. 6. CONCLUSIONS This paper describes the identification of the operational requirements for the public education sector procurement in the Maldives. The operational requirements involve legal and practical constraints of public sector procurement. In order to identify the operational requirements a field research was conducted involving focus groups and analysis of official tendering and BEC documentation. Based on the field research, it is evident that the public sector has legal and practical difficulties to adopt many of the available MCDA methods for bid evaluation. For instance, having price and delivery duration as evaluation criteria as compulsory by regulation for every bid and the formal obligation of announcing allocated weights for every criterion in advance causes the rejection of any method that requires automatic allocation of weights. As BEC members are holding senior posts in the public sector, a simple error free method which consumes minimum time to evaluate bids is crucially important in practice. As a consequence of the findings of this paper and as the result of criteriabased evaluation on operational requirements against the comprehensive list of MCDA methods presented in Appendix 1, TOPSIS and COPRAS emerged as suitable methods for the e-Procurement DSS in the Maldivian context. As future work and follow-up to the research presented in this paper, it would now be interesting to see if the findings presented are transferable to other contexts of the Maldivian public sector procurement. REFERENCES Aghajani, M., & Hadi-Vencheh, A. (2011). A multiple attribute decision making model in the presence of grey numbers. Paper presented at the 3rd International Conference on Advanced Management Science, Singapore. Amponsah, C. T. (2011). Application of multi-criteria decision making process to determine critical success factors for procurement of capital projects under publicprivate partnerships. International Journal of the Analytic Hierarchy Process, 3(2), 107-129.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

224

Adil, M., Nunes, M. B., Peng, G. C.

Arrowsmith, S., & Anderson, R. D. (Eds.). (2011). The WTO regime on government procurement: challenge and reform. Cambridge: Cambridge University Press. Baily, P., Farmer, D., Jessop, D., & Jones, D. (1994). Purchasing Principles and Management (7th ed.). London: Pitman Publishing. Barbour, R. S., & Kitzinger, J. (Eds.). (1999). Developing focus group research: Politics, theory and practice. London: SAGE. Belton, V., & Stewart, T. J. (2002). Multiple Criteria Decision Analysis: An Integrated Approach. Dordrecht Kluwer Academic Publishers. Bottani, E., & Rizzi, A. (2006). A fuzzy TOPSIS methodology to support outsourcing of logistics services. Supply Chain Management: An International Journal, 11(4), 294308. Brown, R. B., Wright, R. D. J., Cloke, C. G., Morris, T. B., & Trumper, I. F. S. (1984). Government purchasing : a multi-department review of government contract and procurement procedures. London: HMSO. Chamodrakas, I., Leftheriotis, I., & Martakos, D. (2011). In-depth analysis and simulation study of an innovative fuzzy approach for ranking alternatives in multiple attribute decision making problems based on TOPSIS. Applied Soft Computing, 11(1), 900-907. Chen, S., Osman, M., & Peng, G. C. (2012). Information systems evaluation: methodologies and practical case studies. In P. Isaias & J. M. B. Nunes (Eds.), Information systems research and exploring social artifacts: Approaches and methodologies. Hershey, USA: IGI Global. Dhaulathuge Maaliyyathu Gaanoonu 2006 (Maldives). Dhaulathuge Maaliyyathuge Gavaaidhu 2009 (Maldives). England, W. B. (1967). The purchasing system. Homewood, Illinois: Irwin. Falagario, M., Sciancalepore, F., Costantino, N., & Pietroforte, R. (2012). Using a DEA-cross efficiency approach in public procurement tenders. European Journal of Operational Research, 218(2), 523-529. Flick, U. (2009). An introduction to qualitative research (4th ed.). London: SAGE. Freeman, R. E. (1984). Strategic management: A stakeholder approach. Boston: Pitman. Guitouni, A., & Martel, J.-M. (1998). Tentative guidelines to help choosing an appropriate MCDA method. European Journal of Operational Research, 109(2), 501521. Huang, S.-l. (2011). Designing utility-based recommender systems for e-commerce: Evaluation of preference-elicitation methods. Electronic Commerce Research and Applications, 10(4), 398â&#x20AC;&#x201C;407. Jahanshahloo, G. R., Lotfi, F. H., & Davoodi, A. R. (2009). Extension of TOPSIS for decision-making problems with interval data: Interval efficiency. Mathematical and Computer Modelling, 49(5-6), 1137-1142. Krueger, R. A. (1998). Analyzing and reporting focus group results. London: Sage. Leenders, M. R., & Fearon, H. E. (1997). Purchasing and supply management (7th ed.). Chicago: Irwin. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

225

sector e-procurement decision support system

Lloyd-Evans, S. (2006). Focus groups. In V. Desai & R. B. Potter (Eds.), Doing development research (pp. 153-162). London: Sage publication. Min, H. (1994). International supplier selection: A multi-attribute utility approach. International Journal of Physical Distribution & Logistics Management, 24(5), 24-33. Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). London: SAGE. Moshkovich, H., Mechitov, A., & Olson, D. (2005). Verbal decision analysis. In J. Figueira, S. Greco & M. Ehrgott (Eds.), Multiple criteria decision analysis : state of the art surveys (pp. 609-637). Boston: Springer. Olson, D. L. (2004). Comparison of weights in TOPSIS models. Mathematical and Computer Modelling, 40, 721-727. Opricovic, S., & Tzeng, G.-H. (2004). Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS. European Journal of Operational Research, 156(2), 445-455. Rowlinson, S., & McDermott, P. (Eds.). (1999). Procurement Systems: A guide to best practice in construction. London: E & FN Spon. Sanayei, A., Mousavi, S. F., Abdi, M. R., & Mohaghar, A. (2008). An integrated group decision-making process for supplier selection and order allocation using multiattribute utility theory and linear programming. Journal of the Franklin Institute, 345(7), 731â&#x20AC;&#x201C;747. Triantaphyllou, E. (2000). Multi-criteria decision making methods : a comparative study. Dordrecht ; Boston, Mass.: Kluwer Academic Publishers. Tsaur, R.-C. (2011). Decision risk analysis for an interval TOPSIS method. Applied Mathematics and Computation, 218(8), 4295â&#x20AC;&#x201C;4304. Weele, A. J. v. (2000). Purchasing and supply chain management : analysis, planning and practice (2nd ed.). London: Business Press. Weele, A. J. v. (2010). Purchasing and supply chain management : analysis, strategy, planning and practice (5th ed.). Andover: Cengage Learning.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

226

Adil, M., Nunes, M. B., Peng, G. C.

APPENDIX 1

1 2 3 4 5

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

Table 1. MCDA methods (Adapted from Guitouni and Martel (1998)) Method Author(s) Linear weighting and elementary methods Weighted Sum Churchman and Ackoff (1954) Lexicographic method Roy and Hugonnard (1982) Conjunctive method Hwang and Youn (1981) Disjunctive method Chen and Hwang (1992) Maximin method Hwang and Youn (1981) Single synthesizing criterion or utility theory TOPSIS Hwang and Youn (1981) MAVT Keeney and Raifa (1976) UTA Jacquet-Lagreze and Siskos (1982) SMART Edwards (1971) MAUT Bunn (1984) AHP and ANP Saaty (1980), Saaty (2005) DEA Talluri et al. (1999) COPRAS Zavadskas et al. (2007); Chatterjee et al. (2011) Outranking methods ELECTRE De Boer et al. (1998); Dulmin and Mininno (2003) ELECTRE I Roy (1968) ELECTRE IS Roy and Bouyssou (1993) ELECTRE II Roy and Bertier (1971) ELECTRE III Roy (1978) ELECTRE IV Roy and Hugonnard (1982) ELECTRE TRI Yu (1992); Mousseau et al. (2000) PR OMETHEE Dulmin and Mininno (2003) PROMETHEE TRI Figueira et al. (2004) PROMETHEE/GAIA technique Dulmin and Mininno (2003) NAIADE Munda (1995) ELECCALC Kiss et al. (1994) UTADIS Doumpos et al. (2001) MELCHIOR Leclerc (1984) ORESTE Roubens (1980) REGIME Hinloopen and Nijkamp (1982) PROMSORT Araz and Ozkarahan (2007) EVAMIX Voogd (1983) QUALIFLEX Paelinck (1978) Fuzzy methods Fuzzy relationship hierarchy Lin and Chen (2004) Fuzzy set approach Sarkar and Mohapatra (2006) Fuzzy suitability index (FSI ) Bevilacqua et al. (2006) Fuzzy weighted sum Baas and Kwakernaak (1977) Fuzzy miximini Bellman and Zadeh (1970) Ng and Skitmore (1995); Vokurka et al. (1996); Kwong et AI methods al. (2002); Choy et al. (2002); Choy et al. (2003); Choy et al. (2005) CBR Ng and Skitmore (1995); Choy et al. (2003) Mixed methods Martel and Zaras method Martel and Zaras (1990); Martel and Zaras (1995) Fuzzy conjunctive/ disjunctive Dubois, Prade and Testemale (1988) method

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

Identifying operational requirements to select suitable decision models for a public

227

sector e-procurement decision support system

APPENDIX 2 Operational requirements identified for the purpose of criteria-based evaluation are as follows: 1. The maximum tender for the evaluation is MVR1.500,000.00. 2. The minimum tender for the evaluation is MVR25.000.00. 3. Different cost bands are evaluated differently. 4. Public announcement should be made for every procurement costing most then MVR25.000.00. 5. There are a minimum of two criteria for evaluation. 6. There can be more criteria for evaluation based on the procurement. 7. Allocation of criteria and weights are based on the needs of the organisation. 8. A pre-bid meeting is compulsory and it needs to be announced. 9. Specification should be provided to potential bidders during the pre-bid meeting. 10. Marking criteria with weights are provided in advance in pre-bid meeting. 11. All required documents should be submitted with the bid and the requirements need to be informed to bidders. 12. If any bidder requires, calculations procedures are explained. 13. All bids are submitted on specific date and time. All the documents are checked verified during the submission process. 14. It requires minimum three BEC members to evaluate bids. 15. Basis for evaluation solely depends on the information provided in pre-bid meeting. 16. Suppliersâ&#x20AC;&#x2122; bids need to be verified for correct information. 17. Suppliersâ&#x20AC;&#x2122; previous jobs are evaluated based on available information. 18. Submitted support documents are primary source of information and they are assessed. 19. Assess the bid price compare to the expected work. 20. Suppliersâ&#x20AC;&#x2122; performances are evaluated based on the criteria provided and according to the weights and marks in allocated schemes provided in advance. 21. Marks are allocated based on the criteria and weights provided during pre-bid meeting in relation to performances of suppliers. 22. Technical expertise is used to get advice and explanations on procurement of technical good and services. 23. A through check is made if the proposed goods or services meet the specified standard. 24. Every criterion is assessed independently from one another. 25. All the criteria need to be evaluated. 26. No ranking can be made in evaluation; rather, marks are allocated in evaluation. 27. Pair wise comparison cannot be done. 28. In evaluation stage no changes to criteria, weights and requirements should be made. 29. Incomplete bids should be rejected. 30. Evaluation calculations are shown to bidders if requested. 31. BEC needs to approve of the winner. Evaluation analysis does not grand awarding the bid to the winner. 32. BEC need to state the reason for selection of the specific bid. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

228

Adil, M., Nunes, M. B., Peng, G. C.

33. Bidders are informed the winner but not marks. 34. If any bidder wants more clarification, evaluation calculations are shown. 35. No discrimination in evaluation. 36. Evaluation method needs to be accurate. 37. Evaluation method should be using reasonable amount of resources and provide reasonable results. 38. Evaluation method should comply with procurement rules and regulations. 39. Evaluation method should provide no chance of manipulation from both sides. 40. Evaluation method needs to help minimise complaints. 41. Evaluation method needs to support utility concept. 42. Evaluation method should be clear and easily understandable.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.211-228

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 229-256 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200002

THE USEFULNESS OF USER TESTING METHODS IDENTIFYING PROBLEMS ON UNIVERSITY WEBSITES

Layla Hasan Department of Computer Information Systems, Zarqa University, Jordan _____________________________________________________________________ ABSTRACT This paper aims to investigate the usefulness of three user testing methods (observation, and using both quantitative and qualitative data from a post-test questionnaire) in terms of their ability or inability to find specific usability problems on university websites. The results showed that observation was the best method, compared to the other two, in identifying large numbers of major and minor usability problems on university websites. The results also showed that employing qualitative data from a post-test questionnaire was a useful complementary method since this identified additional usability problems that were not identified by the observation method. However, the results showed that the quantitative data from the post-test questionnaire were inaccurate and ineffective in terms of identifying usability problems on such websites. Keywords: Usefulness, user testing, methods, university websites, usability, comparison

1. INTRODUCTION The root of usability is in the field of Human Computer Interaction (HCI), which is a broad field related to all the aspects and ways in which people interact with computers (Stone et al., 2005). Humans interact with computers through a user interface. The design of this user interface and, specifically, the usability of the interface is a core area in the field of HCI (Gray and Salzman, 1998). The concept of usability has been defined and measured differently by different authors. For example, Nielsen (2003) stated that usability is not a single attribute; instead, usability is defined in terms of five characteristics: learnability, efficiency, memorability, errors and satisfaction. This definition indicates that usability is defined in terms of a set of attributes or design _____________________________________________________________________________________________ Manuscript first received/Recebido em: 01/01/2014 Manuscript accepted/Aprovado em: 24/06/2014 Address for correspondence / Endereço para correspondência Layla Hasan, Department of Computer Information Systems, Zarqa University, Jordan E-mail: l.hasan2@yahoo.co.uk Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

230 Hasan, L.

goals of a system/product. However, the International Standards (ISO 9241-11, 1998) provide a broader definition of usability, stating that: “Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO 9241-11, 1998). Usability Evaluation Methods (UEMs) are a set of methods used to evaluate human interaction with a product; they are aimed at identifying issues or areas of improvement in this interaction in order to increase usability (Gray and Salzman, 1998). A variety of usability evaluation methods have been developed to identify usability problems. These methods have been categorized differently by different authors. For example, Hasan et al. (2012) categorized usability evaluation methods into three categories in terms of how usability problems were identified: user-based UEMs (which involve real users in the process of identifying usability problems), evaluatorbased UEMs (which involve evaluators in the process of identifying usability problems), and tool-based UEMs (which involve the use of software tools and models in the process of identifying usability problems). The user testing methods (user-based UEMs) aim to record users’ performance while interacting with an interface and/or users’ preferences or satisfaction with the interface being tested. The most common user-based UEM relates to user testing. The other methods are either variations of a user testing approach or supplementary techniques that could be used with a user testing method. Section two summarizes the most common user-based UEMs. Earlier research has employed different user testing methods in the evaluation of the usability of university websites, such as observations and/or questionnaires (Tüzün et al., 2013; Chaparro, 2008; Alexander, 2005; Kasli and Aucikurt, 2008; Christoun et al., 2006; Lencastre and Chaves, 2008; Mustafa and Al-Zoua'bi, 2008; Hasan, 2014). The results of these studies were useful in providing an idea regarding common usability problems that were identified on university websites. However, there is a lack of research that compares issues identified by different user testing methods in the evaluation of the usability of university websites in terms of the types of usability problem that can be identified by them. The aim of this research is to investigate the usefulness of three supplementary user testing methods (employing observation, quantitative data from the post-test questionnaire, and qualitative data from the post-test questionnaire) regarding their ability or inability to identify specific usability problems on university websites. The specific objectives of this research are: 1. To employ three supplementary user testing methods (observation, quantitative data from post-test questionnaire, and qualitative data from post-test questionnaire) to evaluate the usability of three selected university websites; 2. To illustrate the types of specific usability problem identified by each method; 3. To compare the specific usability problems identified by each method with the other methods; and 4. To provide empirical evidence regarding the usefulness of each of the three methods in evaluating the usability of university websites. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

231

This paper is organized as follows. Section two summarizes user-based usability evaluation methods while section three reviews earlier research that has used user testing methods in the evaluation of the usability of university websites. Section four describes the methods used. Section five outlines the main results and section six discusses the results in the light of the literature. Finally, section seven reviews and presents some conclusions. 2. USER-BASED USABILITY EVALUATION METHODS (UEMs) This section reviews the most common user testing methods that can be used to evaluate the usability of websites. The effectiveness of the various user evaluation methods is also presented. a) User Testing The user testing method is “a systematic way of observing actual users trying out a product and collecting information about the specific ways in which the product is easy or difficult for them” (Dumas and Redish, 1999). It is the most important and useful approach since it provides direct information regarding how real users use the interface and it illustrates exactly what problems the users encounter in their interaction (Nielsen and Mack, 1994). Different supplementary techniques have been suggested for use during a user testing session, such as making different types of observation (e.g., notes, audio, video or interaction log files) to capture users’ performance; questionnaires and interviews have also been suggested as ways of collecting data concerning users’ satisfaction (Nielsen, 1993; Rogers et al., 2011; Dumas and Redish, 1999; Rubin, 2008). Capturing user performance can be automated using tools such as Camtasia and/or Morae. Camtasia is a screen capture software package, provided by the TechSmith Company, which has proved to be an effective tool for capturing website usability data (Goodwin, 2005). Morae software, also provided by the TechSmith Company, is used to gain a deep understanding of customers’ experience through its ability to record user interactions with an application, site, or product (TechSmith, 2014). Morae records desktop activity on the user’s computer and makes a camera video of the user. Using TechSmith Corporation’s patented Rich Recording Technology, the Morae recorder captures all system events, including the screen, mouse clicks, web page changes, onscreen text, any notes or markers logged by observers and more. Also, Morae can capture the participant’s face with a Web cam via a Picture-in-Picture mode. b) Think-Aloud Method This is a user testing method with a condition: the condition of asking users to think aloud during their interaction with an interface (Lazar, 2006; Nielsen, 1993). Nielsen (1993) indicated that having users verbalize their thoughts using this method offers an understanding of how users view or interpret an interface. However, the Think-Aloud method has some disadvantages related to the fact that the test setting, with an observer and recording equipment, does not represent a natural setting; this therefore will not encourage users to act and talk naturally (van den Haak and de Jong, 2005).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

232 Hasan, L.

c) Constructive Interaction (also known as Co-Discovery Learning) This method is a think-aloud method with one condition: the condition of having two users (instead of a single user) interacting with an interface together or working together to complete specific tasks (Holzinger, 2005; Nielsen, 1993). The main advantage of employing this technique is that the test situation is much more natural in comparison with the Think-Aloud tests because people are used to verbalizing their thoughts when trying to solve a problem together (Holzinger, 2005; Nielsen, 1993). However, the unnatural settings which are associated with the Think-Aloud method constitute one of the drawbacks of the constructive interaction method. d) Questionnaires and Interviews Different types of questionnaire (e.g., closed or open) and interviews (e.g., unstructured, semi-structured or structured) are considered useful and simple techniques that collect data regarding users’ satisfaction with, or preferences for, a user interface, such as the features and the presentation of websites (Bidgoli, 2004; Rogers et al., 2011; Rubin, 2008). These could be used as supplementary techniques to the user testing method or they could be used alone. However, if these techniques are used alone then they are considered as indirect usability methods because they do not study the user interface directly; instead, they reflect users’ opinions about that interface (Holzinger, 2005; Nielsen, 1993). Various satisfaction/usability questionnaires have been found in the literature that can be used as post-test questionnaires for assessing the usability of and/or satisfaction with websites. Examples of common satisfaction/usability questionnaires are:  System Usability Scale (SUS): This is a common, reliable, simple, low-cost usability scale that can be used for global/general assessments of subjective assessments of a system’s usability (Brooke, 1996; Sauro, 2011). SUS was developed by the Digital Equipment Corporation and consists of 10 questions that are scored on a 5-point Likert scale on strength of agreement (Tullis and Stetson, 2004). SUS yields a single number representing a composite measure of the overall usability of the system being studied. SUS scores have a range of 0 to 100 (Brooke, 1996). Its ease of administration and scoring makes it a popular choice among usability professionals (Finstad, 2006).  Questionnaire for User Interaction Satisfaction (QUIS): This was developed at the University of Maryland and was designed to evaluate users’ satisfaction with different aspects of an interface (Chin et al., 1988). QUIS consists of demographic, overall system satisfaction and interface questions on terminology; screen factors and system feedback; system capabilities and learning factors; as well as questions on system components such as technical manuals, online tutorials, multimedia, voice recognition, virtual environments, Internet access, and software installation (Harper et al., 1997). Each question measures users’ perceptions on a 9-point categorical scale. Additional space is provided to allow users to make comments on the interface factors. QUIS is available in two different formats: a long form with 71 questions and a short form with 27 questions. However, Tullis and Stetson (2004) made a modification to the short form to make it appropriate to websites.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

233

They dropped three questions (e.g., “Remembering names and use of commands”). 

Computer System Usability Questionnaire (CSUQ): This is a short and reliable questionnaire that was developed by IBM ( Lewis, 1995). CSUQ is composed of 19 questions. Each question is a statement with a rating on a seven-point scale of “Strongly Disagree” to “Strongly Agree” (Tullis and Stetson, 2004). CSUQ measures three factors: system usefulness, informational quality, and interface quality.

 Software Usability Measurement Inventory (SUMI): This was developed by the Human Factors Research Group (Kirakowski and Corbett, 1993). SUMI measures users’ attitudes to specific software systems (Kirakowski, 1995). It consists of 50 attitude statements, to each of which the user may respond “agree, don’t know, or disagree” and provides an overarching usability rating across five subscales (Veenendall, 1998). The five subscales are: efficiency, effect, helpfulness, control and learnability.  End-User Computing Satisfaction Questionnaire (EUCS): This was developed by Doll and Torzadeh in 1988. EUCS is a short, reliable, valid and easy to administer questionnaire that can be used as a general measure of user satisfaction with all types of applications (Doll and Torkzade, 1988). EUCS has 12 items that measure five components of end-user satisfaction: content, accuracy, format, ease of use, and timeliness. The questionnaire uses a fivepoint Likert scale.  Website Analysis and MeasureMent Inventory (WAMMI): This is a web analytics service that measures and analyses user satisfaction with a website (Alva et al., 2003). It is the best tool to assess user experience as it relates to the actual experience visitors have of a website (WAMMI, 2014). WAMMI was created by Nigel Claridge and Jurek Kirakowski; it consists of 20 statements, which were selected from a large range of questions about users' experiences with websites. It is based on five factors: attractiveness, controllability, efficiency, helpfulness and learnability. e) Eye Tracking This is a method which aims to record and observe exactly the paths on a computer screen users follow while using the web (Nielsen and Pernice, 2010). In order to employ this method, special eye tracking equipment is needed to be built into the computer monitor. Also, eye tracking software is required to track the user’s screen. This method is useful because it observes normal behavior and can discover exactly what users look at. However, the collected data cannot provide information regarding whether users are happy or confused when they look at certain things on a screen and not at others (Nielsen and Pernice, 2010). Also, the special equipment required to conduct eye tracking method is expensive.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

234 Hasan, L.

3. LITERATURE REVIEW This section summarizes earlier research that evaluated the usability of university websites using different types of user-based usability evaluation methods. For example, some studies employed more than one user testing methods, including observation, in the evaluation of the usability of such websites. Examples of such include the study conducted by Tüzün et al. (2013) who employed five user testing approaches related to observation, questionnaires, interviews, think-aloud, and eye tracking to evaluate the usability of the registration unit sub-site at Hacettepe University. The results showed that the students experienced several usability problems related to the design of the tested sub-site, including: inappropriate design of pages (e.g. long pages), inappropriate design of the menus, poor organization of the sub-site's data, and inappropriate font size (small). Chaparro (2008) also employed three user testing methods (observation, questionnaires and interviews) to evaluate the usability of a university portal website. The results showed that the website had several usability problems including: having non-obvious links, inappropriate organization of information on the site, inappropriate use of images (icons), and inappropriate page design (having three different search functions on the same page). Similarly, Alexander (2005) employed three user testing methods (observation, think-aloud and questionnaire) in an evaluation of the usability of 15 university websites. The results highlighted six usability problems that were found on the websites including: poor information architecture, poor content (e.g., insufficient detail), ineffective internal search function, difficulty using PDF documents, poor page design, and broken links. Other studies were found in the literature that employed only the questionnaire method in the evaluation of university websites. Kasli and Aucikurt (2008), for example, employed only the questionnaire method and asked 54 students to investigate 132 websites of tourism departments at universities in Turkey. The results shed light on the following usability problems: old information, lack of an internal search function, lack of support for foreign languages, and failure to display important information (e.g., an academic calendar, FAQs and programs). Christoun et al. (2006) also investigated students’ overall satisfaction with an academic website with regard to its technology, usability, aesthetics and content using an online questionnaire. The results showed that the website had usability problems related to: ineffective search function and difficulty in finding information. Similarly, Lencastre and Chaves (2008) employed only the questionnaire method aimed at students in the evaluation of the usability of an educational website used by Master’s degree students at Minho University, Portugal. The results showed that in general the website had positive ratings. However, Lencastre and Chaves (2008), unlike the other studies which involved the questionnaire method in the evaluation of the usability of university websites, provided evidence regarding the inaccuracy of the results that were obtained using the questionnaire method. For example, they stated that many respondents via their answers to the questionnaire indicated that there was no online help on the site while in fact it was always there.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

235

Also, the study conducted by Mustafa and Al-Zoua'bi (2008) employed a questionnaire to evaluate the usability of nine Jordanian university websites by 252 students. The results showed that the usability of the tested websites was, in general, good. However, the study of Mustafa and Al-Zoua'bi (2008) provided no details regarding specific types of usability problem that users identified on university sites. Furthermore, Hasan (2014) asked 237 students to provide ratings for nine Jordanian university websites using usability criteria. The students were also asked to report qualitatively what they liked and disliked regarding the design of the websites. The results showed that the most common weaknesses on the websites included: the design of the sites, the fact that most of the sites were inconsistent in terms of colors and fonts, the Arabic and English language interfaces, the design of the pages, and the lack of support for the Arabic language. The studies outlined in this section showed that earlier research employed different types of user-based usability evaluation methods in the evaluation of the usability of university websites and proved the usefulness of these methods in highlighting common usability problems that could be found on such websites from the studentsÂ´ point of view of. The results from those studies outlined above, which employed more than one method in the usability evaluation, stressed the fact that employing more than one user testing method is useful since the methods complement each other and provide a clear picture of usability problems on the tested websites. Unfortunately, these studies did not provide detailed information regarding to what extent the different methods complemented each other in the identification of the usability problems. No examples were offered by these studies regarding common usability problems that were identified by the methods, and/or unique issues that were raised by each of the employed methods. This research aims to address this gap noted in the literature by illustrating the extent to which three supplementary user testing methods (observation, and using both quantitative and qualitative data from the satisfaction questionnaire) complement each other in the evaluation of the usability of university websites. This research provides empirical evidence regarding the specific usability problems that were identified by each of the three methods. 4. METHODOLOGY In order to employ and compare usability methodologies with regard to university websites, three university websites in Jordan were selected. The selection was based on the results of one of the major international university ranking websites, which is Eduroute (Eduroute, 2011). The three universities, which had the highest rankings provided by Eduroute for the year 2011, were selected as the sample for this research. The universities were: Hashemite University, the University of Jordan, and Yarmouk University. These universities are large public universities offering a variety of disciplines. To achieve the aim of this research, three common supplementary user testing methods were used. The first related to the observation; different types of observation were used, including the observer taking notes and using Camtasia software, to capture performance data. The second method related to the closed-ended post-test

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

236 Hasan, L.

questionnaire, which was designed to collect quantitative data to assess users’ satisfaction with the tested sites. The third method related to the open-ended post-test questionnaire, which was designed to collect qualitative data to assess users’ satisfaction with the tested sites. A pre-test questionnaire was developed to gather users’ background information. A task scenario was developed for each of the three websites, as shown in Table 1. This included typical tasks for the three university websites that represented their actual use. The typical tasks were derived from an earlier study, which listed types of pages visited frequently by Jordanian students on a university website (Hasan, 2013). For clarity and simplicity throughout this paper, the titles website 1, website 2 and website 3 are used to refer to the University of Jordan’s website, the Hashemite University’s website, and Yarmouk University’s website, respectively. Table 1: Task scenarios for the three websites Task No.

Website

Tasks

Task 1

All websites

When will the summer semester for the academic year 2011/2012 start?

Task 2

All websites

What is the subject of the university’s latest announcement?

Task 3

All websites

What activities/student clubs are supported by the university?

Task 4

All websites

What services are offered by the university to the students?

Website 1

What are the conditions/requirements to transfer from a public university to the University of Jordan?

Website 2

What are the conditions/requirements to transfer from a public university to Hashemite University?

Website 3

What are the conditions/requirements to transfer from a public university to Yarmouk University?

Website 1

What is the time(s) for the Principles of Management Information Systems course, which will be offered by the Faculty of Business for the summer semester 2011/2012?

Website 2

What is the time for the Management Information Systems course, which will be offered by the Faculty of Economics and Administrative Sciences for the summer semester 2011/2012?

Website 3

What is the time for the Knowledge Management course, which will be offered by the Faculty of Economics and Administrative Sciences for the summer semester 2011/2012?

All websites

Is there a study plan for the mathematics course at the Bachelor level?

Website 1

What is the academic rank of Dr. Mohammad Al-Kilani, who is a member of the Chemistry Department’s staff?

Website 2

What is the academic rank of Dr. Ayman Issa, who is a member of the Chemistry Department’s staff?

Website 3

What is the academic rank of Dr. Ahmad Al Omari, who is a member of the Chemistry Department’s staff?

Task 5

Task 6

Task 7

Task 8

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

237

In order to gather quantitative and qualitative data from the students regarding their preferences with the tested websites, three specific post-test questionnaires were developed for the purpose of this research, despite the fact that there are many validated satisfaction/usability scales that are available (for examples the ones mentioned in section 2). This is related to the fact that all of the usability questionnaires reviewed in section 2 were developed to measure the overall usability of a system or to measure general usability issues. Thus, they are not comprehensive enough to determine specific usability issues with educational websites. None of them were developed specifically to evaluate the usability of educational websites. Furthermore, the literature review presented above shows that those questionnaires were not used to evaluate the usability of educational websites. Therefore, none in their current state could adequately assess the usability problems specific to an educational website. Each user responded to the appropriate post-test questionnaire after interacting with each website. The post-test questionnaires were divided into two sections. The first was designed to collect quantitative data using closed-ended questions (Appendix 1), while the second section was designed to collect qualitative data using open-ended questions. The first section was based on specific criteria for evaluating the usability of educational websites that was proposed in an earlier study (Hasan, 2014). This part of the questionnaire included 25 questions which were scored on a 7-point Likert scale on strength of agreement. The questionnaire was organized into five sections: navigation, architecture/organization, ease of use and communication, design, and content. The navigation section consisted of six questions which assessed whether the site included the main tools (e.g. navigation menu, internal search facility) and links which facilitated the navigation of users through the site, enabling them to reach the required information quickly. The architecture/organization section consisted of two questions. These related to the structure of a site's information which should be divided into logical, clear groups; each group should include related information. The ease of use and communication section consisted of four questions, which related to the cognitive effort required to use a website, and to the existence of basic information which facilitated communications with the university in different ways. The design section consisted of six questions that related to the visual attractiveness of the site's design; the appropriateness of the design of the site's pages; and the appropriateness of the use of images, fonts and colors in the design of the site. Finally, the content section consisted of seven questions which assessed whether the information on the site was up-to-date, sufficient, relevant to user needs, and accurate; and whether a site included the information students required (e.g. information about the university, information about the colleges, and information about the departments). The second part of the questionnaires included two open-ended questions. The first asked the students to list five features on each site they most preferred, while the second question asked the students to list five features they disliked on each site. Thirty students were recruited to conduct this research. The students were selected randomly from different faculties at Zarqa University in Jordan where the researcher was working. The students reported that they had not explored the three websites prior to the usability testing. All user testing sessions followed the same

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

238 Hasan, L.

procedure. Data were gathered using screen capture software (Camtasia) with four questionnaires and observations of the users working through the tasks. The user session began with the researcher welcoming the user and asking him/her to fill out a pre-test questionnaire in order to obtain information regarding his/her background and experience. Before beginning the tasks related to each website, the user was asked to explore the website for a maximum of 10 minutes. After the exploration, the user was given the tasks for a particular website from the three tested sites. As the user worked on each task, the observer noted the sequence of pages, the time taken to complete each task, and any comments made by the user. After completing the tasks for the tested website, the user was given the post-test questionnaire to fill out in order to get his/her feedback. Then the user took a break before beginning to test the second website. A similar procedure was followed by the user while testing the second and third sites. At the end, the user was thanked. For each session, the order of the three websites that were evaluated was changed so that each website was tested fairly by all the users since, while testing the first website, the user might be slow and unfamiliar with the testing tasks. The user testing methods were deployed between May 2012 and June 2012. No website changes were made by the universities’ web designers during this timeframe. Descriptive analysis was used to analyze data collected from the pre-test questionnaires. The performance data were summarized in two ways: the tasks’ timings (in seconds) and the tasks’ accuracy. Descriptive statistics were used to obtain the mean time (in seconds) and the standard deviation for each of the eight tasks (Appendix 2). Furthermore, the one-way within subject ANOVA test was employed for each of the eight tasks. The results of this test are presented in Appendix 3. The task accuracy represented the percentage of users who completed each task successfully within the time benchmark. In order to identify the usability problems from the performance data, two steps were used, as suggested by Rubin (2008): a) Identifying the problematic tasks: all the problematic tasks were considered. b) Identifying students’ problems and conducting a source of error analysis. In order to identify users’ problems/obstacles/difficulties with the problematic tasks, and to investigate the usability problems behind these, different sources were examined. These included the in-session observation notes, notes taken from reviewing the ninety Camtasia sessions, and users’ comments noted during the test. Data collected from the post-test questionnaires were used to find evidence of usability problems with the sites. Likert scores were calculated for each statement in section 1 of the post-test questionnaire for each site in order to obtain the results concerning the participants’ satisfaction with the sites. A Likert score of 1-3 was regarded as a negative response, 5-7 a positive response, and 4 a neutral one. Qualitative data obtained from students’ responses to the open-ended questions in the post-test questionnaires were taken into account in determining the usability problems. Users’ answers were translated into English from Arabic and were then combined for each site; these were grouped under the five categories of the post-test questionnaires and their corresponding sub-categories.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

239

5. RESULTS This section presents the findings obtained from the analysis of the different user testing methods. The first subsection presents an overview of the users in terms of their characteristics while the second subsection presents the findings from the performance data and observations. The third and fourth subsections present the quantitative and qualitative results obtained from the analysis of the post-test (satisfaction) questionnaires, respectively. 5.1 Participants’ Characteristics The participant students comprised 16 males and 14 females. They were in general experienced computer and web users; 97% had more than three years of computer experience and 63% had used the Internet for more than three years. The students were also frequent users of the website of their university; 97% of them had used their university’s website. 5.2 Performance Data and Observation The summary of the performance data is presented in Appendix 2 and Table 2. Appendix 2 presents the mean time in seconds and the standard deviation for each task.

Table 2 presents the levels of accuracy for each task across the three websites. It shows that all the tasks were problematic; no task was performed successfully by all the students on any of the three websites. This indicates that it was difficult for the students to perform the typical tasks on the three websites. However, Table 2 shows that the tasks’ accuracy varied for each website. A one-way within-subjects ANOVA test showed that the time spent performing the majority (six) of the eight tasks was significantly different for the three sites. Appendix 3 shows the results of the ANOVA test for each task. Table 2: Task accuracy Accuracy Score Task Website 1

Website 2

Website 3

Task 1

73%

13%

87%

Task 2

93%

67%

93%

Task 3

30%

10%

60%

Task 4

13%

43%

63%

Task 5

83%

73%

83%

Task 6

13%

57%

63%

Task 7

10%

80%

Task 8

83%

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

240 Hasan, L.

The observation notes, the notes generated from reviewing the 90 Camtasia files, and usersâ&#x20AC;&#x2122; comments from the user testing were summarized in terms of tasks. This observation summary presents a snapshot of each task and highlights the critical incidents that occurred during each task across the sites. These incidents represent potential problems with usersâ&#x20AC;&#x2122; interactions with the sites. Using the observation summary, a large number of usability problems were identified on each website for each task. Similar problems in each site were grouped together to generate a list of problems for each site. The three lists generated were examined to identify similar problems across the three sites. Consequently, 19 common areas of usability problems were identified, which suggested identifying 19 problem sub-themes. These 19 problem sub-themes suggested identifying six problem themes based on the type of problems identified. The six problem themes related to: navigation, design, content, internal search, ease of use and support for the Arabic language, and missing capabilities. The 19 usability problems were then classified into two types based on their seriousness. The first type was major problems and related to mistakes/errors that a student made and was unable to recover from in order to complete the task on time. The user might or might not have realized his/her mistake/error. The second type related to minor problems, which included mistakes/errors that a student made but was able to recover from and complete the task, or other difficulties that were observed or indicated by usersâ&#x20AC;&#x2122; comments while performing the tasks. Consequently, 12 major usability problems and seven minor usability problems were identified on the websites. Table 3 shows the 19 problem sub-themes, their themes, and the seriousness of each problem. Table 3: Usability problem themes and sub-themes that students identified by observation, together with their seriousness. Problem Theme

Navigation

Design

Problem Sub-theme

Seriousness of the Usability Problem

Links were not obvious

Major

Links not opening the destination pages (e.g., it remained on the same page)

Minor

Misleading links (e.g., the link name did not match the content of the destination page)

Major

Weak navigation support (e.g., lack of navigational menu or links to other pages in the site)

Minor

Broken links

Major

Inappropriate choice of fonts (e.g., small size)

Minor

Inconsistency in the language of the interface (e.g., links at the English interface opened pages that displayed Arabic content and vice versa)

Major

Misleading images (e.g., it did not have a link when it was suggested to users that it had one)

Minor

Inappropriate page design (e.g., long and cluttered pages)

Major

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

241

The Usefulness of user testing methods in identifying problems on University Websites

Ineffective text format (e.g., the information was not aligned correctly on the siteâ&#x20AC;&#x2122;s pages)

Minor

Broken images

Minor

No information regarding the type of a file that a link would open

Major

Irrelevant content (e.g., the content of a page was not clear to users)

Major

Old content

Major

Empty pages

Major

Internal Search

Ineffective internal search (e.g., it did not work properly)

Major

Ease of Use and Support for the Arabic Language

Difficult interaction with a website

Major

Not supporting the Arabic language

Major

Missing Capabilities

Missing functions/ information (e.g., no internal search facility)

Minor

Content

The following presents an explanation and examples regarding the major and minor usability problems that were identified on the websites. The 12 major usability problems related to: 1. Links were not obvious: this problem was identified on website 1 and related to links that were not situated in an obvious location on a page so they could be recognized by students. For example, the link required to change the language of the Home page from English (default) into Arabic was located at the very top righthand corner of the Home page, as shown in Figure 1. It was observed that this link was not obvious for 20 (out of the 30) students. The observer helped the 20 students to find this link so that they were able to continue the other tasks. It was difficult to interact with the site for most of the students in the English language interface. 2. Misleading links: this related to the fact that the destination page, which was opened by a link, was not expected by students because the link name did not match the content of the destination page. This problem was found on websites 1 and 3. An example of such a link is the Home Page link which was located in the top menu of the Registration unit sub-site in the two websites. The students expected this link to open the Home Page of the university. However, it opened the Home Page of the sub-site (Registration unit sub-site). 3. Broken links: some broken links were identified on websites 1 and 2, which prevented the students from performing the required tasks successfully. An example of these links was the Study Plans link, which was located at the new student subsite in website 2. 4. Inconsistency in the language of the interface: this problem was mainly related to the existence of links in the Arabic interface that opened pages with English content. This problem was identified on websites 1 and 2.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

242 Hasan, L.

5. Inappropriate page design: this problem related to pages on the tested websites that did not clearly represent their content; it was found on all three websites. For example, the Self-Registration page in the Registration unit sub-site for website 1, which included the Course Schedule link, was very long and cluttered, as shown in Figure 2. It was observed that most of the students (26 out of 30) (Table 2) could not find the Course Schedule link on this page and therefore could not complete the required task successfully (Task 6).

Figure 1: The Home page of website 1

6. No information regarding the type of file that a link will open: this problem was found on websites 1 and 2, and related to the fact that there was no information regarding the type of file (e.g., PDF) that a link would open. For example, the Calendar link in the advertisements sub-site on website 2 opened the academic calendar of the university in PDF format. It was observed that the PDF file took a long time to open, which made most of the students think that there was a problem with the link. Most of the students stated qualitatively that there was no file to be opened and therefore, they did not complete the related task successfully (Task 1). 7. Irrelevant content: this problem related to the fact that some pages on the tested websites displayed an unclear message. This problem was found on all three of the tested websites (e.g., the Course Schedule page on website 3). 8. Old content: this problem related to the fact that the content of a page was out of date. This problem was found on website 2. The academic calendar in the Registration unit sub-site displayed old information: the calendar for the year 2010.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

243

This problem was the main reason behind the failure to complete Task 1 successfully on website 2 by most of the students (26 out of 30) (Table 2).

Figure 2: The upper part of the Self-Registration page in the Registration unit sub-site for website 1

9. Empty pages: this problem related to the existence of some pages on website 2 that had empty content. Examples on these pages are: the Services page on the Registration sub-site, and the Clubs and Activities page on the Student Affairs subsite. 10. Ineffective internal search: this problem related to the fact that the internal search function on all three websites did not work properly. 11. Difficult interaction with a website: this problem related to the fact that it was not easy to visit some pages on the three websites. For example, it was observed that it was difficult for the students to visit the Registration unit sub-site from the Home page of the three websites (Table 2). 12. Not supporting the Arabic language: this problem related to the fact that websites 1 and 2 did not support the Arabic language. For example, the Home page of website 2, including its main menu, was displayed only in the English language. Also, the faculties and departments sub-sites at these two universities did not display their content in the Arabic language; they were presented only in English. It was observed that this problem prevented most of the students from completing many of the required tasks successfully (e.g., Tasks 6, 7 and 8 at website 1; Tasks 1, 3, 4, 5, 6, 7 and 8 at website 2).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

244 Hasan, L.

The seven minor usability problems that were identified by the observation method related to: 1. Links not opening the destination page: this problem related to a link that did not work properly as expected by the students; it remained on the same page. For example, the Student Gate link, which was located on the Home page of website 1, had this problem. 2. Weak navigation support: this problem related to a page that did not have a navigational menu or links to other pages on the site. This problem was found on the Self Registration page at the Registration unit sub-site on website 1. 3. Inappropriate choice of fonts: this problem related to the use of an inappropriate font size (small size) at websites 1 and 3. For example, the observation showed that the font used for the Arabic link on the Home page of website 1, which was used to change the language of the interface from English into Arabic, was small from the viewpoint of the students (Figure 1). 4. Misleading images: this problem related to the existence of some images on website 1 which did not function as the students expected. For example, such images did not have a link while users expected them to have one (e.g., the logo of the University at the header of the Graduate Studies sub-site). 5. Ineffective text format: this problem related to the existence of some pages on website 2 in which the information, figures and tables were not aligned correctly. Examples included these pages: the Financial Department unit sub-site and the Forms page on the Registration sub-site. 6. Broken images: this problem related to the existence of some images at the Clubs at the Student Affairs sub-site on website 2 which were broken. 7. Missing functions/information: this problem related to the fact that website 2 did not have some functions or capabilities. For example, it did not have an internal search facility at the Registration unit sub-site. 5.3 The Quantitative Data from the Post-Test Questionnaires The analysis of the quantitative data from the post-test questionnaires showed that websites 1 and 2 had usability problems (Appendix 1). The results showed, however, that the students were satisfied with website 3; the Likert scores for all the statements of the post-test questionnaire for this site were more than 5. It is important to explain the reason behind the studentsâ&#x20AC;&#x2122; satisfaction with website 3, which was clearly identified by the observation. The reason was the support of website 3 for the Arabic language. The negative statements with Likert score ratings of 1 to 3 (Appendix 1, statements 1, 3, 4, 7, 10, 11, 12, 19, and 25) identified six usability problems on websites 1 and 2. Each of these problem sub-themes were compared with the problem sub-themes that were identified by the observation method for agreement. Consequently, these statements were mapped to the identified problem themes and subthemes. It was found that four of the six problems were also identified by the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

245

observation method; these related to: misleading links, old content, ineffective internal search, and difficult interaction with a website. However, two statements (Appendix 1, statements 7, 25) identified two new problem sub-themes, relating to poor structure and missing information, respectively. The poor structure problem suggested a new problem theme, which concerns architecture/ organization, while the missing information problem was mapped to the content problem themes. The new problems, as well as their descriptions, are shown in Table 4. Table 4: New problem themes and sub-themes that were identified by the quantitative data of the posttest questionnaires, together with their descriptions. Problem Theme

Problem Sub-theme

Content

Missing information (Adequate information about the departments of the university was not displayed)

Architecture/organization

Poor structure (complicated)

However, the results showed that the quantitative data obtained from the analysis of the post-test questionnaires revealed that this method was inaccurate and ineffective in identifying usability problems on university websites. The inaccuracy issue of this method was discovered by making a comparison between the results of this method and the results of the observation. It was found that, despite the fact that this method identified similar types of usability problems (problem sub-themes) that were also identified by the observation method, there were clear differences between the location of three (out of four) of the problems obtained using this method and the location of the usability problems that were identified by the observation method. Specifically: 1. The observation method identified misleading links as major problems on websites 1 and 3. However, the quantitative data of the post-test questionnaires identified this kind of problem on websites 1 and 2. 2. The observation method showed that the three websites had ineffective internal search problems as major problems. However, the post-test data of the questionnaires identified this problem only on websites 1 and 2. 3. The observation method identified difficulty interacting with the three websites as major problems, while the quantitative data of the post-test questionnaires identified this type of problems only on websites 1 and 2. Regarding the inefficiency issue of the quantitative data obtained from the analysis of the post-test questionnaires, it was found that this method was unable to identify several usability problems on the tested websites. The post-test questionnaires included specific statements which related to potential usability problems on the tested websites, including: broken links, inappropriate font size, inappropriate page design, irrelevant content, and empty pages (Appendix 1, statements 5, 15, 17, 20, 21, 23, 24 and 25). However, it was found that the students rated these statements as either

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

246 Hasan, L.

positive or neutral when most of these types of usability problem were identified by the observation method as major issues (Table 3). 5.4 The Qualitative Data from the Post-Test Questionnaires The analysis of the qualitative data from the post-test questionnaires resulted in the identification of 16 usability problems. These problems were compared and then mapped to the appropriate problem themes and sub-themes identified by the previous two methods (observation, and the quantitative data from the satisfaction questionnaires). It was found that there were no matches between five problems and the identified problem sub-themes. Therefore, five new sub-themes were identified and mapped to four appropriate problem themes (navigation, design, architecture/ organization, and ease of use and support for the Arabic language). Table 5: New problem themes and sub-themes that were identified by the qualitative data of the post-test questionnaires, together with their descriptions. Problem Theme Navigation

Problem Sub-theme Orphan pages (pages that did not have any links ) Unaesthetic design (unattractive interface)

Design

Inappropriate choice of colors (e.g. inappropriate combination of background and link colors)

Architecture/Organization

Deep architecture (the number of clicks to reach goals was more than 3)

Ease of Use and Support for the Arabic Language

Slow downloading of the siteâ&#x20AC;&#x2122;s pages

The unique identification of the five usability problems by the qualitative data obtained from the post-test questionnaires provided additional evidence regarding the inefficiency of the quantitative data obtained from the analysis of the post-test questionnaires. Despite the fact that there were five specific statements in the post-test questionnaire relating to the five new problems that were uniquely identified by the qualitative method (Appendix 1, statements 6, 8, 9, 13 and 16), the students did not rate these statements negatively and therefore did not identify these problems via the quantitative data. Regarding the 11 usability problems which were also identified by the other previous methods, it was found that there were similarities between most of them (seven out of the 11) and the usability problems that were identified by the observation method in terms of their locations. Specifically: ď&#x201A;ˇ

This method identified broken links on website 2, which was identified also as a major problem by the observation method.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

247



The students via this method qualitatively identified usability problems related to inappropriate choices of font on websites 1 and 3; this was also identified by the observation method.



The students identified four other usability problems on the three websites relating to inappropriate page design, irrelevant content, ineffective internal search, and difficult interaction with the websites. These problems were also identified by the observation method as major problems on the three websites (Table 3).



This method identified a lack of support for the Arabic language on websites 1 and 2, which was also identified by the observation method on those websites as a major problem.

However, it was found that there were differences between four (out of 11) of the usability problems that were identified by the qualitative data of the questionnaire method, and the usability problems that were identified, either by the observation or the quantitative data of the post-test questionnaire, in terms of their location. Specifically: 

This method identified misleading links on websites 2 and 3, while the observation method identified misleading links on websites 1 and 3. However, the quantitative data of the post-test questionnaire identified the misleading link problem on websites 1 and 2.



The students via this method identified inconsistency in the language on websites of 2 and 3, while the observation method identified this problem on websites 1 and 2.



The students qualitatively identified the problem of empty pages on websites 1 and 2, while the observation method identified this as a major problem only on website 2.



The missing information problem was identified by this method on websites 1 and 2, while it was identified by the quantitative data of the post-test questionnaire only on website 2.

6. DISCUSSION Researchers need to understand the usefulness or the contribution of the most common supplementary user testing methods (e.g., observation, and using quantitative and qualitative data from post-test questionnaire) in identifying specific usability problems on a university website, in order to decide which method to use when evaluating such websites. This research has uniquely addressed a specific gap in the literature regarding illustrating the extent to which three supplementary user testing methods (observation, and using both quantitative and qualitative data from a satisfaction questionnaire) to complement each other in the evaluation of the usability of university websites. This was achieved by conducting a comparison among the results obtained by each of the three user testing methods, which uncovered the contribution or the value of each of the three methods in the identification of specific usability problems on such websites. Table 6 presents a comparison of the output of the three methods.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

248 Hasan, L.

Table 6: Comparison of the output of the three methods.

Problem Theme

Problem Sub-Theme Links were not obvious Links not opening the destination pages Misleading links

Navigation

Weak navigation support Broken links Orphan pages Inappropriate choice of fonts Inconsistency in the language of the interface Misleading images Inappropriate page design

Design

Ease of Use and Support to Arabic Language

website 1

Observation

websites 1 and 3 website 1 websites 1 and 2

websites 2 and 3 NA

website 2

websites 1 and 2

websites 1 and 3 websites 1 and 2 website 1 websites 1, 2 and 3

NA NA

websites 1 and 2 websites 1 and 3 websites 2 and 3

NA NA

websites 1, 2 and 3

website 2

Broken images

website 2

Unaesthetic design

Inappropriate choice of colors

NA websites 1, 2 and 3 websites 1, 2 and 3

No information regarding the type of a file that a link will open

websites 1 and 2

websites 1, 2 and 3

Old content

websites 1, 2 and 3 website 2

website 2

Empty page

website 2

Missing information Internal Search

Qualitative Data from the Post-test Questionnaire NA

Ineffective text format

Irrelevant content Content

website 1

Quantitative Data from the Post-test Questionnaire NA

Ineffective internal search Difficult interaction with a website Not supporting the Arabic language Slow downloading of the siteâ&#x20AC;&#x2122;s page Poor structure Deep architecture

websites 1, 2 and 3 websites 1, 2 and 3 websites 1 and 2

websites 1 and 2 websites 1 and 2

NA NA

websites 1 and 2 websites 1 and 2 websites 1, 2 and 3 websites 1, 2 and 3 websites 1 and 2 websites 1, 2

NA website 2 NA Architecture/ organization NA NA websites 1, 2 Missing Missing functions website 2 NA NA Capabilities NA: not applicable, which means that the method was not able to identify the problem.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

249

It is worth mentioning that Table 6 provides evidence regarding the fact that if just one method is employed to identify usability problems on university websites, it may result in costly developments that may be prove to be misguided. The following illustrates how each set of data provided by each method could be interpreted:  The data obtained from the observation method provided clear indications that all three tested websites had usability problems. Furthermore, as explained earlier, the seriousness of the problems were also determined, which could facilitate giving higher priority to major problems while giving less priority to minor problems during the development of the websites. However, Table 2 shows that this method was unable to identify seven usability problems which were identified by the students using the other methods. These included: slow downloading of the site’s pages, poor structure, deep architecture, missing information, inappropriate choice of colors, unaesthetic design, and orphan pages.  The data obtained from the quantitative data from the post-test questionnaire provided indications that only websites 1 and 2 had a small number of usability problems. However, the location of these problems was not clear. This would require further efforts to be made by the developer in order to find out which usability problems to fix. Furthermore, the quantitative data obtained from the post-test questionnaire indicated that website 3 had no usability problems when in fact it had major usability problems (e.g. ineffective text format, difficulty interacting with the website), which were identified using the observation methods.  The qualitative data obtained from the post-test questionnaire indicated that the three websites had usability problems and identified usability problems which were not identified by the other methods. However, Table 2 shows that this method was not able to identify specific major usability problems that were identified by the other methods (e.g. links were not obvious, there was no information regarding the type of file that a link would open) Despite the fact that this research focused mainly on addressing a unique gap that was found in the literature regarding uncovering the role of three supplementary user testing methods in evaluating the usability of university websites, there was agreement between most of the results of this research and the results of earlier research, which employed a variety of user testing methods while evaluating the usability of university websites (Table 7). Such agreement highlighted common usability problems that were identified on various university websites from the viewpoint of students which should be taken into consideration when evaluating or developing such websites. Table 7 summarizes the common usability problems that were identified in earlier research which were also identified in this research.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

250 Hasan, L.

Table 7: Common usability problems on university websites from students’ viewpoints.

Usability Problem

Reference(s)

Inappropriate page design

Tüzün et al. (2013); Chaparro (2008); Alexander (2005); Hasan (2014)

Inappropriate font size

Tüzün et al. (2013)

Non-obvious links

Chaparro (2008)

Inappropriate organization of information on a site

Chaparro (2008)

Deep architecture

Alexander (2005)

Poor content (e.g. lack of clarity, long pages, insufficient details)

Alexander (2005); Kasli and Aucikurt (2008)

Ineffective internal search

Alexander (2005); Christoun et al. (2006)

Difficulty using PDF format documents

Alexander (2005)

Broken links

Alexander (2005)

Old information

Kasli and Aucikurt (2008)

Lack of internal search function

Kasli and Aucikurt (2008)

Lack of support for the Arabic language

Hasan (2014)

Difficulty finding information

Christoun et al. (2006)

Inconsistency (e.g. colors, fonts, design of the pages)

Hasan (2014)

The results of this research, which has made a contribution in terms of highlighting the usefulness of each of three user testing methods regarding their ability to identify usability problems on university websites, stress the usefulness of the observation method in identifying specific usability problems on such websites. This method was the most effective method compared to the other two in terms of the number of usability problems identified and the seriousness of the problems. This method identified 19 specific types of usability problem on university websites; 12 (out of the 19) were major usability problems which related to mistakes/errors that a student made from which he/she was unable to recover and complete the task on time. However, seven (out of the 19) were minor usability problems which included mistakes/errors that a student made but was able to recover from and complete the task, or other difficulties that were observed, or users’ comments while performing the tasks. Section 5.2 summarizes the 19 specific usability problems, which related to six problem areas that were identified by this method. This method, however, could not identify problems related to seven areas: missing information, poor structure, orphan pages, unaesthetic design, inappropriate choice of colors, deep architecture, and slow downloading of the site’s pages. However, the results of this research showed that the quantitative data from the post-test questionnaires method was inaccurate and ineffective in identifying usability problems on university websites. These results were in agreement with the results provided by earlier research (Lencastre and Chaves, 2008) regarding shedding light on

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

251

the inaccuracy of the questionnaire method. However, Lencastre and Chaves (2008) in their research employed only the questionnaire method in the evaluation of the usability of an educational website, while this research employed three user testing methods, and compared the results obtained from the analysis of the three methods. The results of this research suggest that the quantitative data from the post-test questionnaires was not effective or useful in pointing out specific usability problems on university websites. The results of this research also showed that the quantitative data obtained from the post-test questionnaire method reflected the studentsâ&#x20AC;&#x2122; overall satisfaction with a site. This result agreed with the indications provided by researchers regarding the effectiveness of quantitative data in highlighting usersâ&#x20AC;&#x2122; overall preferences with regard to a site (Holzinger, 2005; Nielsen, 1993). However, this research provided empirical evidence regarding the inability of quantitative data from the post-test questionnaire method to complement the other user testing methods in terms of identifying specific types of usability problem on university websites. Conversely, the findings of this research suggested the usefulness of using openended questions in the post-test questionnaire to identify additional and specific usability problems on university websites, problems which could not be identified using the observation method. Furthermore, this research illustrated the types of specific usability problem that students could identify using this method after their interaction with a university website. Section 5.4 summarizes the five usability problems which were identified by this method; these were related to four main problem areas and were not identified by the observation. This research showed that qualitative data from the post-test questionnaire method have the ability to complement other user testing methods in terms of identifying additional specific usability problem on university websites, which could not be identified by the other methods. The results of this research suggested that the observation method should be employed, followed by open-ended questions using a post-test questionnaire since they complemented each other and resulted in identifying a large and specific number of usability problems while evaluating the usability of university websites. Regarding collecting quantitative data using a post-test questionnaire, this research suggests using a small number of questions if the researcher(s) wishes to discover the overall preferences/satisfaction with a website.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

252 Hasan, L.

7. CONCLUSIONS This research has achieved its aim and illustrated the usefulness of three supplementary user testing methods (observation, and using both quantitative and qualitative data from satisfaction questionnaires) regarding their ability or inability to identify specific usability problems on university websites. This research confirms the complementary value of two user testing evaluation methods: observation and qualitative data from the post-test questionnaires; each is capable of identifying usability problems which the other is unlikely to identify. This research also provided empirical evidence regarding the inability of the quantitative data of the post-test questionnaire to complement the other methods and identify specific usability problems on university websites. A final Likert-style questionnaire provided at the end of the evaluation is likely to result in an overall impression of satisfaction with a site, but will not reveal the exact differences between the sites; most certainly it cannot be used alone to make design recommendations This research has managerial and academic implications. Managerial implications: this research provides empirical evidence for the selected university websites regarding weak design issues on their websites. Also, it is expected to raise awareness among universities regarding how to investigate and improve the usability of their websites by clarifying explicitly the role of specific user testing methods in identifying usability problems. Academic implications: this paper presents an evaluation of the value/contribution of three user testing methods in the evaluation of the usability of university websites, and illustrates what kind of problems each method is capable or not capable of identifying. Also, it provides empirical evidence regarding the usefulness of these methods in the evaluation of the usability of university websites. The results of this research could contribute to the literature regarding the effectiveness of supplementary user testing methods in complementing each other while they are being used to evaluate the usability of websites. However, this research has certain limitations. The sample used in this research was limited to students of only one university in Jordan. Also, other stakeholders of the selected university websites (e.g. faculty staff, employees, parents) were not taken into consideration while conducting the user testing methods. It is worth mentioning that, despite the fact that this research was conducted in Jordan, where the selected websites displayed Arabic content, it is likely that the results can be useful to other countries because many of the details concerning the specific usability problems may be experienced by users on other university websites in various countries. This was shown in the discussion section which presents an agreement between most of the results of this research and the results of earlier research; it, therefore, highlighted common usability problems that were identified on various university websites from the studentsÂ´ viewpoint. Acknowledgment: This research was funded by the Deanship of Research and Graduate Studies in Zarqa University /Jordan.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

The Usefulness of user testing methods in identifying problems on University Websites

253

REFERENCES Alexander, D. (2005). How Usable are University Websites? A Report on a Study of the Prospective Student Experience. Technical Report. Monash University. Alva, M., Martínez, A., Cueva, J.M., Sagástegui, C., and López, B. (2003). Comparison of Methods and Existing Tools for the Measurement of Usability in the Web. In the Proceedings of the 3rd International Conference on Web Engineering (ICWE’03), Spain, Springer, 386-389. Bidgoli, H. (2004). The Internet Encyclopaedia. John Wiley and Sons. Brooke, J. (1996). SUS: A Quick and Dirty Usability Scale. In: P.W. Jordan, B. Thomas, B.A. Weerdmeester & I.L. McClelland (Eds.), Usability Evaluation in Industry. London: Taylor and Francis. Chaparro, B. (2008). Usability Evaluation of a University Portal Website. Usability News, 10(2), 1-7. Chin, J.P., Diehl, V.A. and Norman, K.L. (1988). Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface. In: Soloway E, Frye D, Sheppard SB, editors. In the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; Washington, DC. New York, ACM Press, 213-221. Christoun, S., Aubin, H., Hannon, C., and Wolk, R. (2006). Web Site Usability in Higher Education. Information Systems Education Journal, 4(110). Doll, W.J. and Torkzadeh. G. (1988). The Measurement of End-user Computing Satisfaction. MIS Quarterly, 12(2), 259-274. Dumas, J. S. and Redish, J. C. (1999). A Practical Guide to Usability Testing, Second Intellect Ltd, Rev Sub Edition. Finstad, K., (2006). The System Usability Scale and Non-Native English Speakers. Journal of Usability Studies, 1(4), 185-188. Goodwin, S. (2005). Using Screen Capture Software for Website Usability and Redesign Buy-in. Library Hi Tech, 23(4), 610-621. Gray, W. and Salzman, C. (1998). Damaged Merchandise? A Review of Experiments that Compare Usability Evaluation Methods. Human-Computer Interaction, 13, 203261. Eduroute. (2011). http://www.eduroute.info. [accessed 14.02.2011]. Harper, B., Slaughter, L., and Norman, K. (1997). Questionnaire Administration via the WWW: A Validation & Reliability Study for a User Satisfaction Questionnaire. In: Lobodzinski S, Tomek I, editors. In the Proceedings of the World Conference on the WWW, Internet & Intranet, Toronto, CA. Charlottesville (VA), AACE, 808-818. Hasan, L. (2014). Evaluating the Usability of Educational Websites Based on Students' Preferences of Design Characteristics. The International Arab Journal of e-Technology (IAJeT), 3(3), 179-193. Hasan, L. (2013). Heuristic Evaluation of Three Jordanian University Websites. Informatics in Education, 12(2), 231–251. Hasan, L., Morris, A., and Probets, S. (2012). A Comparison of Usability Evaluation Methods for Evaluating E-Commerce Websites. Behaviour & Information Technology Journal, 31(7), 707–737.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

254 Hasan, L.

Holzinger, A. (2005). Usability Engineering Methods for Software Developers. Communications of the ACM, 48(1), 92-99. ISO 9241-11. (1998). International Standard. First Edition, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs). Part11: Guidance on Usability. http://www.idemployee.id.tue.nl/g.w.m. rauterberg/lecturenotes/ISO9241part11.pdf. [accessed 03.04.2010]. Kasli, M. and Avcikurt, C. (2008). An Investigation to Evaluate the Websites of Tourism Departments of Universities in Turkey. Journal of Hospitality, Leisure, Sport and Tourism Education, 7(2), 77-92. Kirakowski, J. and Corbett, M. (1993). SUMI: The Software Measurement Inventory. British Journal of Education Technology, (24), 210-212. Kirakowski, J. (1995). The Software Usability Measurement Inventory: Background and Usage, in Usability Evaluation in Industry, Jordan, P., Thomas, B. and Weerdmeester, B. (eds.), Taylor & Frances, London. Lazar, J. (2006). Web Usability: A User-Centered Design Approach. Pearson/Education Inc. Lencastre J. and Chaves J. (2008). A Usability Evaluation of Educational Websites. in the Proceedings of EADTU Conference. France. Lewis, J.R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human Computer Interaction, 7(1), 57-78. Mustafa, S. and Al-Zoua'bi, L. (2008). Usability of the Academic Websites of Jordan's Universities. In the Proceedings of the International Arab Conference on Information Technology. Tunisia. Nielsen, J. (1993). Usability Engineering. London: Academic Press. Nielsen, J. (2003). Usability 101: Introduction to Usability. Useit.com, http://www.useit.com/alertbox/20030825 .html. [accessed 14.02.2011]. Nielsen, J. and Mack, R. L. (1994). (Eds.). Usability Inspection Methods, John Wiley & Sons. New York. Nielsen, J. and Pernice, K. (2010). Eyetracking Web Usability. New Riders Press. ISBN 0-321-49836-4. Rubin, J. (2008). Handbook of Usability Testing: How to Plan. Design, and Conduct Effective Tests. Wiley Publishing Inc. Rogers, Y., Sharp, H., and Preece, J. (2011). Interaction Design: Beyond HumanComputer Interaction. Wiley. Third Edition. Sauro, J., (2011). Measuring Usability with the System Usability Scale (SUS). https://www.measuringusability.com/sus.php#comments. [accessed 24.07.2014]. Stone, D., Jarrett, C., Woodroffe, M., and Minocha S. (2005). User Interface Design and Evaluation. The Open University. Morgan Kaufmann. TechSmith. (2014). http://www.techsmith.com/morae.html. [accessed 22.07.2014]. Tullis, T. S. and Stetson, J. N. (2004). A Comparison of Questionnaires for Assessing Website Usability, In the Proceedings of the Usability Professionals Association (UPA) 2004 Conference, Minneapolis, USA. Tüzün, H., Akinci, A., Kurtoğlu, M., Atal, D., and Pala, F. (2013). A Study on the Usability of a University Registrar’s Office Website Through the Methods of Authentic

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

255

The Usefulness of user testing methods in identifying problems on University Websites

Tasks and Eyetracking. The Turkish Online Journal of Educational Technology (TOJET), 12(2), 26-38. Van den Haak, M. and de Jong, M. (2005). Analyzing the Interaction between Facilitator and Participants in Two Variants of the Think-Aloud Method. In the Proceedings of IEEE International Professional Communication Conference, 323-327. Veenendall Van E. (1998). Questionnaire based Usability Testing. In: Unknown, editor. EURO & Y2K: The Industrial Impact. In the Proceedings of the the European Software Quality Week, Brussels, BE. San Francisco (CA): Software Research, Inc. 1-9. WAMMI. (2014). http://www.wammi.com. [accessed 22.07.2014]. Appendix 1. Likert scores and standard deviation of the post-test questionnaires. No. Navigation

Question

Moving around the website without getting lost was easy 2 It was easy to go to the home page from any sub page of the site 3 The site’s internal search function was effective 4 Links are working properly and not misleading so that the user knows what to expect from the destination page 5 The site has no broken links 6 The site has no orphan (dead-end pages) Architecture/organization 7 The structure of the site (organization of the site’s information) was clear 8 The architecture of the site was not too deep (the number of clicks to reach goals was not too large: e.g. it does not require clicking more than 3 links) Ease of use and communication 9 Quick downloading of the site’s pages 10 Interaction with the website was easy 11 Finding the required information was easy 12 Generally, I was satisfied with the ease of use of the site Design 13 The site was attractive and appealing so that it impresses the potential student 14 The use of images was appropriate 15 The size of the text made the site easy to read 16 The choice of colors was appropriate 17 The design of the pages was appropriate 18 Page layout or style was consistent throughout the web site Content 19 The information presented on the site was up-to-date 20 The information on the site was clear and relevant to user needs 21 There were no ‘under construction’ pages 22 The information on the site was accurate 23 Adequate and clear information about the university was displayed 24 Adequate and clear information about the faculties was displayed 25 Adequate and clear information about the departments was displayed STD: Standard Deviation

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

Website 1

Website 2

Website 3

Likert Score

STD

Likert Score

STD

Likert Score

STD

3.96

1.57

2.66

3.99

5.9

6.16

4.14

2.73

5.25

4.58

6.42

7.89

3.21

4.63

3.23

3.40

5.10

3.93

2.34

3.89

2.45

5.97

5.88

4.45 4.83

3.29 3.76

4.32 5

2.51 3.68

6.23 5.83

8.66 6.26

4.21

3.13

3.55

3.02

6.07

5.40

4.69

3.48

4.5

3.04

5.87

5.44

4.31 3.38 3.07

1.98 2.12 3.45

4.77 2.73 2.59

2.51 3.45 3.24

5.97 6.17 5.65

6.68 6.02 4.58

3.13

2.53

4.15

5.73

4.96

4.66

2.85

4.3

2.93

5.37

3.95

4.41 5.31 5.41 4.97

3.73 4.22 3.48 2.15

4.86 5.42 5.6 4.59

2.97 4.65 4.89 3.21

5.4 6.27 5.77 5.48

3.35 6.52 4.79 3.93

4.79

2.54

4.59

2.37

5.53

4.61

5.87

6.60

3.2

4.19

6.6

9.18

4.71

3.34

4.2

2.50

6.37

6.80

5.29 4.72

4.47 3.13

4.87 4.93

3.68 2.63

5.48 5.83

4.79 5.62

5.41

4.15

4.35

2.51

6.53

8.68

4.90

3.45

4.07

1.80

6.53

7.61

4.46

2.99

3.87

1.80

6.42

7.52

www.jistem.fea.usp.br

256 Hasan, L.

Appendix 2: Mean time (in Seconds) for each task across the three

Task 1

Website 1 Mean Std. Deviation 138.3000 112.6952

Website 2 Mean Std. Deviation 159.3333 82.3664

Mean 79.8333

Task 2

25.2000

22.7390

23.9667

30.6667

21.0333

20.9868

Task 3

130.5667

77.3758

96.5000

64.1640

111.5333

73.9211

Task 4

101.4667

85.0471

54.6000

51.7838

61.2333

65.2918

Task 5

101.5333

83.8104

50.4667

59.7464

69.5333

52.1951

Task 6

172.2000

75.7948

85.3667

56.2320

124.0333

74.8159

Task 7

154.1667

82.3508

123.6333

73.3666

93.4667

107.8475

Task 8

137.0333

66.6170

118.2000

54.9472

75.0000

38.1621

Task

Website 3 Std. Deviation 73.0716

Appendix 3: Result of One-Way within-Subjects ANOVA test for each task among the three sites Task Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8

ANOVA Test (One-Way within-Subjects) Was there a statistically significant difference among Website1, 2 and 3 Yes F(2,87) = 6.152, p=.003 No F(2, 87) = .217, p=.805 No F(2, 87) = 1.685, p=.191 Yes F(2, 87) = 4.083, p=.020 Yes F(2, 87) = 4.500, p=.014 Yes F(2, 87) = 11.743, p=.000 Yes F(2, 87) = 3.484, p=.035 Yes F(2, 87) = 10.213, p=.000

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.229-256

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 257-276 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200003

A FUZZY MULTICRITERIA APPROACH FOR IT GOVERNANCE EVALUATION Angel Cobo University of Cantabria, Santander, Spain Adolfo Alberto Vanti Universidade do Vale do Rio dos Sinos, Brasil Rocío Rocha University of Cantabria, Santander, Spain _____________________________________________________________________ ABSTRACT This work seeks to provide a new multi-criteria approach to assess IT Governance (ITG) in the area of Strategic Alignment. The complete methodological development process is described. The evaluation model uses Fuzzy Analytic Hierarchy Process (FAHP) and it is targeted to IT processes, more specifically to the COBIT© IT maturity levels, domains and processes, thus providing a differentiated analysis of importance for each item. Its relevance is related to addressing isolated and individual evaluation criteria that are normally practiced in audits of processes. The model allows generating information that extends the guarantees of compliance and corporate governance from different organizations. This research demonstrates that the combined use of multi-criteria decision methodologies and soft computing proves to be particularly suitable for Strategic Alignment such as the focal area of COBIT. The model was applied in a big retail Brazilian company. Keywords: Corporate Governance, IT Governance, Strategic Alignment, COBIT, FAHP.

_____________________________________________________________________________________________ Manuscript first received/Recebido em: 10/10/2013 Manuscript accepted/Aprovado em: 16/03/2014. We thank Fapergs and Fundação Carolina. Address for correspondence / Endereço para correspondência Angel Cobo, Doctor’s Degree in Applied Mathematics, University of Cantabria. Avda. Los Castros s/n 39005 Santander (Spain). Email: acobo@unican.es Adolfo Alberto Vanti, Doctor’s Degree in Business Administration. Universidade do Vale do Rio dos Sinos. Escola de Gestão de Negócios. Av. UNISINOS, 950, Cristo Rei, 93022000 - São Leopoldo, RS - Brasil,: Email: avanti@unisinos.br Rocío Rocha, Doctor’s Degree in Business Administration, University of Cantabria. Avda. Los Castros s/n 39005 Santander (Spain). Email: rochar@unican.es Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

258 Cobo, A., Vanti, A. A., Rocha, R.

INTRODUCTION

The multidisciplinary process of Corporate Governance (CG) and of IT Governance (ITG) (Weill & Ross, 2004; Van Grembergen & Haes, 2004) reduce agency conflicts in aspects such as informational asymmetry and help to reach fundamental principles related to disclosure, compliance, fairness, accountability and transparency (Jensen & Meckling, 1976). These principles are supported by good governance codes aligned with the Information Systems that support business processes, as the codes established by the Cadbury Report (1992), OECD (2011) and IBGC (2012). Information Systems (IS) develop operational and managerial activities in internal control, increasing the guarantee of CG mainly related to the measurement requirement of confidentiality, integrity, availability and compliance. These aspects are present in different evaluation frameworks related to information security processes (Kwok & Longley, 1999; Taylor & Fitzgerald, 2007) and COBIT (ITGI, 2013). This paper focuses specifically on the analysis of the strategic alignment focal area, using the COBIT framework and seeking to reduce the problems between business and IT. The lack of a strategic alignment between IT and business causes competitiveness losses as established by (Hirschheim & Sabherwal, 2001; Weiss & Thorogood, 2011) and also a limited improvement in strategic information systems (SIS) planning (Lederer & Sethi, 1992) that supports the achievement of the organizational objectives (Zviran, 1990), as well as the impacts and performance of the organization (Lederer & Mendelow, 1989; Chen, 2010; Li & Tan, 2013; Tiwana & Konsynski, 2010). These studies emphasize that alignment and IT Governance (ITG) must be studied together, because they are strongly related and complementary concepts. ITG guides the use of IT in the company in strategic control and adds value to business, improving decision-making processes (Van Grembergen & De Haes, 2010; Zarvi´c, Stolze, & Thomas, 2012). In this direction of alignment between business and IT through the ITG, we propose a COBIT based model for assessing IT processes, integrating business and fuzzy aspects that will reduce the limitation detailed above. In Marnewick & Labuschagne (2011) it became obvious that CG fails in the decision of IT projects in organizations. The study considered ITG models such as COBIT, ISO 35000, PRICE2 and PMBoK©. In the case of IT Governance using the COBIT, the companies in the study stated that they did not meet the requirements, postponing goals. Strategic alignment problems between IT and business may arise from critical IT management processes with low levels of maturity. These problems can lead to wrong decisions and not reach the desired compliance, leading to problems of governance and conflicts between owners and agents or executives. Different works evidence problems related to the use of technology during decision making processes (Merali et al, 2012; Benítez et al 2012)

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

A Fuzzy Multicriteria Approach For IT Governance Evaluation

259

In this context, this paper aims to contribute to the improvement of ITG by applying a methodology for fuzzy multi-criteria evaluation, namely FAHP (Chang, 1996), and focuses on assessing the level of strategic alignment according to the COBIT 4.1 framework. This FAHP application expands the traditional IT audit process in which the valuation is produced using maturity levels in a pre-established scale (0 to 5 coming from CMMIÂŠ). The inclusion of FAHP methodology in the proposed model provides greater robustness in qualitative and subjective aspects or even fuzzy or unclear results in the pair to pair evaluation of the audited processes. 2.

A FRAMEWORK FOR IT GOVERNANCE: COBIT

COBIT (Control Objectives for Information and related Technology) is a framework created by the ISACA (Information Systems Audit and Control Association) for information technology management and IT Governance itself (ISACA, 2012), it includes an ontological metamodel of IT Governance framework (Neto & Neto, 2013). The ISACA published the current version, COBIT 5, in 2012. In (De Haes et al, 2013) research questions for future research on enterprise governance of IT and COBIT 5 are proposed and discussed. COBIT 5 reveals new conceptual ideas compared to the previous COBIT 4.1 version (Preittigun et al, 2012), however, in this work COBIT 4.1 was used. This version of COBIT defines a process model that subdivides IT into four domains and 34 processes. It provides best practices across a domain and process-based framework and presents activities in a manageable and logical structure. The business orientation of COBIT consists of linking business goals to IT goals, providing metrics and maturity models to measure their achievement, and identifying the associated responsibilities of business and IT process owners. COBIT focuses strongly more on control and less on execution, and is contextualized in Information Technology Governance (ITG). According to Simonson, Johnson and Ekstedt (2010), it is defined as a technology managed and structured in an organization, providing mechanisms that also contribute to the strategic and IT planning of the organization. COBIT supports IT governance by providing a framework to ensure that IT is aligned with the business, IT resources used responsibly, and IT risks managed appropriately. Finally, performance measurement is an essential aspect for IT governance. In short, IT governance (ITG) is structured around 5 major focus areas that are defined by COBIT. These areas describe the topics that executive managers need to address in order to govern IT within their organizations. According to COBITâ&#x20AC;&#x2122;s executive overview (ITGI, 2013), the five focus areas are: Strategic alignment, focused on ensuring the linkage of business and IT plans; Value delivery, about executing the value proposition; Resource management, engaged in optimal investment; Risk management, focused on risk awareness; and, finally, Performance measurement, tracks and monitors strategy implementation. COBIT 4.1 subdivides IT into 4 domains and 34 processes in line with the responsibility areas of planning, building, running and monitoring, providing an end-toend view of IT. COBIT is a tool for managing IT processes that includes concepts of management, mainly through the business requirements for information:

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

260 Cobo, A., Vanti, A. A., Rocha, R.

confidentiality, availability and integrity in four domains and their interrelations. These domains are briefly described below. Plan and Organize (PO) includes strategies and tactics with the intention of identifying the best way on how IT can contribute to the achievement of the business objectives. Acquire and Implement (AI) is a domain that analyzes the IT solutions that need to be identified, developed or acquired, implemented and integrated into the business process. Deliver and Support (DS) refers to the delivery of the services requested, which includes service delivery, safety and continuity of management, support services for users and data management and operational resources. Finally, Monitor and Evaluate (ME) domain establishes the regular assessment processes to ensure adherence to quality and control requirements. This domain addresses performance management, monitoring of internal control, regulatory compliance and governance. Each domain has a different number of processes involved; in fact, PO has 10 processes; AI has 7; DS has 13; and ME has 4. The complete list of 34 processes is included in the annex. It is important to note that this division into domains and processes allows to infer a hierarchy of criteria that can be used later in the AHP (Analytic Hierarchical Process) methodology applied to this work. In short, the hierarchy that will be used is shown in Figure 1. IT Governance Focus Area

PO Plan and Organize

AI Acquire and Implement

DS Deliver and Support

ME Monitor and Evaluate

PO1

AI1

DS1

ME1

PO2

AI2

DS2

ME2

PO3

AI3

DS3

ME3

PO4

AI4

DS4

ME4

PO5

AI5

DS5

PO6

AI6

DS6

PO7

AI7

DS7

PO8

DS8

PO9

DS9

PO10

DS10 DS11 DS12 DS13

Figure 1. Focus area-domain-process hierarchy defined by COBIT 4.1.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

A Fuzzy Multicriteria Approach For IT Governance Evaluation

261

COBIT 4.1 also includes an assessment framework of processes that defines maturity levels for each process. The model is based on the Capability Maturity Model Integration (CMMIÂŠ). This assessment framework defines a 0-5 scale with an alphanumeric description of each numeric value as it is briefly shown in Table 1. This scale allows an assessment of the degree of development of each process in an organization and the evaluation model provides guidelines for assigning maturity levels. Value 0 1 2 3

Description There is no recognition of the need for internal control. There is some recognition of the need for internal control. Controls are in place but are not documented. Controls are in place and are adequately documented. However, the evaluation process is not documented and employees are aware of their responsibilities for control. 4 Managed There is an effective internal control and a risk management and environment with a formal and documented evaluation of measurable controls, which occurs frequently. A limited, tactical use of technology is applied to automate controls. 5 Optimized Control evaluation is continuous, based on self-assessments and gap and root cause analyses. Employees are proactively involved in control improvements. Table 1. Maturity levels (ITGI, 2013).

Name None Initial Repeatable Defined

METHODOLOGY

Our aim is to construct a model that allows the assessment of IT governance in an organization based on the maturity levels of COBITâ&#x20AC;&#x2122;s processes. To achieve this objective we have used a fuzzy multi-criteria decision methodology: FAHP. The proposed approach is a fuzzy extension of the classical Analytic Hierarchy Process (AHP) multi-criteria technique. This technique is especially useful for obtaining a single assessment value based on different previously selected indicators or criteria (levels of maturity of processes). Furthermore, it also allows us to incorporate subjective appreciations or opinions from the IT manager on the processes that may prove more significant when it comes to measuring this assessment of IT governance. In the AHP, each element in the hierarchy of criteria is considered to be independent of all the others. When there is interdependence among criteria, and extension of AHP known as Analytic Network Process (ANP) would be used. This approach to IT evaluation via COBIT 4.1 is very important because it is normally evaluated in an individually and isolated way, without taking into consideration subjective and fuzzy aspects. It means that the maturity levels are evaluated by choosing grades in the CMMIÂŠ (from 0 to 5) scale, ignoring that each evaluated process (34 all) has a strong influence and is influenced.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

262 Cobo, A., Vanti, A. A., Rocha, R.

3.1. Fuzzy numbers (Zadeh, 1965) introduced the theory of fuzzy sets in 1965 to model the concept of vagueness, characteristic of human thought. Fuzzy numbers allow to face problems in which the criteria are not precisely defined. In fuzzy logic, a statement can not only be true or false, as in classical logic, but also, and moreover, it provides a range of intermediate values between absolute certainty and absolute falsehood. In this context, a fuzzy set is determined by a membership function which determines the degree of certainty with which an element x belongs to the set. A triangular fuzzy number is a special type of fuzzy number whose membership is defined by three real numbers, expressed as (l, m, u), where l is the lower limit, m the most promising and u the upper limit value. The membership function of M=(l,m,u) is given by:  x l  m  l if l  x  m  u x  ( x)   if m  x  u u  m if x  l or x  u 0 

The graphical representation of this function can be seen in Figure 2. µ(x)

Figure 2. Membership function defining the triangular fuzzy number M=(a,b,c). The assumption of triangular fuzzy numbers is a simplification, which can be frequently found in the literature and which facilitates fuzzy arithmetic calculations (Meixner, 2009). It is possible to use the operation laws following Zadeh’s extension principle via this simplification which makes calculations much easier. Given the triangular fuzzy numbers A=(a1,a2,a3) and B=(b1,b2,b3), the basic arithmetic operations are defined as follows: A+B = (a1+b1, a2+b2, a3+b3) A*B = (a1b1, a2b2, a3b3) nA = (na1,na2,na3) for all n>0 It is also possible to define the concepts of opposite -A=(-a3,-a2,-a1) and inverse 1/A=(1/a3,1/a2,1/a1) fuzzy triangular numbers.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

A Fuzzy Multicriteria Approach For IT Governance Evaluation

263

3.2. Fuzzy Analytic Hierarchy Process (FAHP) Analytic Hierarchy Process (AHP) is a multi-criteria decision technique proposed by Saaty (1980) to solve problems of planning needs and management of scarce resources that, in time, has become one of the most widely used techniques in decision making processes on multiple criteria (Chang, 1992). In general, this technique can be applied to problem solving that requires an evaluation and measuring in which different and very often opposed criteria intervene. The main advantages of AHP are the relative ease with which it handles multiple criteria, as well as the fact that it is easy to understand and can effectively handle both qualitative and quantitative data (Markaki, Charilas, & Askounis, 2010). The AHP technique is developed through six key stages: •

Definition of the problem and establish clear objectives and expected results.

•

Deconstruction of a complex problem in a hierarchical structure with elements of decision. At a high level of hierarchy, general objectives and criteria are divided into particular objectives or subcriteria for reaching the lowest level in which the alternatives are located.

•

Carrying out of pair comparisons between decision elements, forming comparison matrices based on the establishment of the relative importance between the factors of each hierarchical level.

•

Checking of the consistency properties of the matrices in order to guarantee that the judgments issued by the decision makers are coherent and consistent.

•

Estimation based on previous matrices of the relative weights of the decision elements for achieving the general objective.

•

Making of an evaluation of the alternatives based on the weights of the decision elements.

The relative importance of the decision elements in AHP is assessed indirectly through a series of pairwise comparisons, in which the decider provides preferences by comparing all criteria and subcriteria with respect to upper level decision elements. AHP uses a 1-9 numeric scale in order to establish priority values aij for each pair of criteria. If the element Ei is preferred to Ej then aij>1. At each level of the criteria hierarchy we obtain an n-dimensional squared matrix, where n is the number of elements or criteria of the level. The reciprocal property aij=1/aji and aii=1 and aij >0 always remains the same. In order to calculate the weights that the AHP model will assign to each criterion there are different alternatives. One of the most common methods, though computationally more complicated, is the calculation of an eigenvector associated to the dominant eigenvalue of the comparison matrix. This value must be proximate to n and is also used to define the consistency index (CI) and the consistency ratio (CR), which allows us to value the appreciation carried out by the decider on coherence when estimating the relative importance of the elements.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

264 Cobo, A., Vanti, A. A., Rocha, R.

Among the advantages of the AHP method, we can cite the facility for incorporating multiple criteria, the possibility of using linguistic variables, as well as the need to carry out an exhaustive analysis of the definition of the values of comparison, which leads to a greater understanding of the problem tackled. However, for complex problems or those with many options, excessive computational effort may be required and a high level of pairwise comparisons. In spite of its popularity, the AHP method is often criticized for its inability to adequately handle the inherent uncertainty and imprecision associated with the mapping of a decision-makerâ&#x20AC;&#x2122;s perception to crisp numbers. A natural way to cope with uncertainty in judgments is to express the comparison ratios as fuzzy sets or fuzzy numbers, which incorporate the vagueness of the human thinking. Therefore, fuzzy AHP (FAHP), a fuzzy extension of AHP, can be used to solve hierarchical fuzzy problems (Van Laarhoven & Pedrycz, 1983; Mikhailov & Tsvetinov, 2004). FAHP applications can be found in diverse areas such as selection of operating systems (Balli & Korukoglu, 2009), recruitment of staff, (Chen, 2009), risk assessment projects in information technology (Iranmanesh et al, 2008), selection of ERP systems (Lien and Chan, 2007). Using the concept of triangular fuzzy numbers, one can obtain a "fuzzy" or diffuse version of the classical AHP. When comparing two elements Ei and Ej, the exact value ratio aij can be approximated with a fuzzy ratio which is represented by a fuzzy triangular number. The construction of a hierarchical model in FAHP is exactly equal to the original AHP. The fuzzy numbers required to form the decision matrix may be determined directly according to the decision maker or may derive from linguistic variables in a verbal scale, which can be then converted into fuzzy numbers using a suitable conversion as shown in Table 2 and Figure 3. In order to construct a positive reciprocal matrix of pairwise comparisons, a full set of n(n-1)/2 comparison judgments are required. The pairwise comparison matrix is constructed as

where

;

and

Intensity of Importance of one criterion over another

AHP: Crisp number

FAHP: Fuzzy number

Equal Weak: moderately more important Strong: significantly more important Very strong: strongly more important Absolute: extremely more important Intermediate values

1 3 5 7 9 2,4,6,8

(1,1,2) (2,3,4) (4,5,6) (6,7,8) (8,9,9) (1,2,3) (3,4,5) (5,6,7) (7,8,9)

Table 2. Triangular fuzzy numbers to construct the pairwise comparison matrices (equivalence between the AHP and FAHP approaches).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

265

A Fuzzy Multicriteria Approach For IT Governance Evaluation

Equal

Weak

Very strong

Strong

Absolute

Figure 3. Fuzzy pairwise comparison scale. The final weights of the decision elements can be calculated using different methods that have been proposed in the literature. One of the most popular methods is the Fuzzy Extent Analysis, proposed by Chang (1996). The steps of Changâ&#x20AC;&#x2122;s extent analysis can be summarized as follows: First step: computing the normalized value of row sums by fuzzy arithmetic operations:

Second step: computing the degree of possibility of

defined as

x and y being the values on the axis of the membership function of each criterion. This expression is equivalently expressed as

where . Using these expressions the degree of possibility of to be greater than all the convex fuzzy numbers is computed as follows:

Third step: defining the priority normalized vector of the fuzzy comparison matrix as:

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

266 Cobo, A., Vanti, A. A., Rocha, R.

Observe that the final normalization weight vector W is a real non-fuzzy vector that can be used to perform an evaluation of the alternatives based on the weights of the decision elements in each hierarchy level. The local priorities represent the relative weights of criteria within a group with respect to their parent in the hierarchy. The global priorities are obtained by multiplying the local priorities of the siblings by their parentâ&#x20AC;&#x2122;s local priorities. 4.

A FAHP MODEL BASED ON COBIT FOR IT GOVERNANCE EVALUATION

With the aim of constructing a model that allows us to measure the governance of TI according to the focus areas defined by COBIT 4.1 and taking as reference the basic stages of the methodology described before, we proceeded to carry out the actions illustrated in Figure 4. Each of the stages will be explained in detail below.

Fuzzy pairwise comparison between criteria and sub-criteria

Consistency analysis of the comparisons

Weights calculation

Strategic alignment

Value delivery

Risk management

Resource management

Performance measurement

IT governance focus areas of COBIT

Criteria identification and hierarchical structure construction

Fuzzy Analytical Hierarchy Process (FAHP)

Definition of the objective

Figure 4. Stages for the construction of a model for evaluating IT governance. 4.1. Definition of objectives The model we have developed aims to achieve a double objective. Firstly, to evaluate the level of performance of the IT governance using maturity levels in COBIT processes, and to use it to try to identify critical aspects in the governance model within organizations in five focus areas. With the aim of checking the effectiveness of the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

A Fuzzy Multicriteria Approach For IT Governance Evaluation

267

model, we sought its practical application in a Brazilian firm that operates in Rio Grande do Sul State. 4.2. Identification of criteria hierarchy In order to achieve the proposed objectives, we decided to select the 34 processes defined by COBIT 4.1. These processes are hierarchically organized as shown in Figure 1. In the first level of the criteria hierarchy we consider the four main domains Di (i=1,2,3,4). Specifically, D1 corresponds to “Plan and Organize”, D2 is “Acquire and Implement”, D3 is “Deliver and Support” and D4 is “Monitor and Evaluate”. COBIT 4.1 also defines a set of processes associated with each main domain; we will use the following notation to refer to the processes involved in Di: Pi,j with i in {1,2,3,4} and j in {1,2,…,ni}, where i indicates the associated domain and j the number of process in the domain. Each domain has a different number of processes involved, in fact n1=10, n2=7, n3=13, and n4=4. The complete list of 34 processes is shown in the Annex. The processes define the second level in the criteria hierarchy. 4.3. Pairwise comparison between criteria COBIT 4.1 defines significance levels for each process, and priority levels associated with each process and focus area. With this information, we proceed to the construction of pairwise comparisons matrices between areas and between the processes associated with the same domain. The building process is described as follows. In accordance with the significance level defined by COBIT for each process, we define the following function:

These importance levels are defined in the mapping of IT processes for IT governance focus areas (Appendix II of COBIT 4.1) COBIT also associates a primary or secondary priority for each process in each focus area. According to this information we define:

Firstly, we proceed to define the fuzzy pairwise comparisons between domains with respect to a particular focus area. For Da and Db domains, we need to compare both of them using the fuzzy preference scale defined in Table 2, which assigns fuzzy

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

268 Cobo, A., Vanti, A. A., Rocha, R.

triangular numbers to different levels of preference with respect to the focus area selected (FAp). In order to define the importance levels the following process we compute:

and

where ceil() is the function that maps a real number to the smallest following integer. After this process the fuzzification comprises the process of transforming the crisp value into a triangular fuzzy number using the correspondence defined in Table 2. In this work we focus on the evaluation of strategic alignment. According to the previous expressions the fuzzy pairwise comparison matrix for this focus area is shown in Table 3. D1 D2 D3 D4 D1 (1,1,2) (3,4,5) (4,5,6) (2,3,4) D2 (1/5,1/4,1/3) (1,1,2) (1,2,3) (1/3,1/2,1) D3 (1/6,1/5,1/4) (1/3,1/2,1) (1,1,2) (1/3,1/2,1) D4 (1/4,1/3,1/2) (1,2,3) (1,2,3) (1,1,2) Table 3. Pairwise domains comparison matrix in â&#x20AC;&#x153;Strategic Alignmentâ&#x20AC;? focus area. These numbers maintain the fuzzyfication from the priority of each process in each focus area of COBIT and the importance levels, defined in the same technological process evaluation framework. Once the comparisons at the first hierarchical level were made, the same process was made comparing the different criteria that stem from the general criteria in the hierarchy. For Da a domain, we need to compare the processes associated to Da using the fuzzy scale. In this case we proceed as follows: For Pa,i and Pa,j processes in the domain Da, we define

Again the fuzzification process is performed using defined in Table 2.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

269

A Fuzzy Multicriteria Approach For IT Governance Evaluation

This expression allows constructing the pairwise comparisons matrices for the specific domains. The Tables 4, 5, 6 and 7 show the matrices associated to the 4 domains in the strategic alignment focus area.

P1,1 P1,2 P1,3 P1,4 P1,5 P1,6 P1,7 P1,8 P1,9 P1,10

P1,1

P1,2

P1,3

P1,4

P1,5

P1,6

P1,7

P1,8

P1,9

P1,10

(1,1,2) (1/4,1/3,1/2) (1/4,1/3,1/2) (1/7,1/6,1/5) (1/4,1/3,1/2) (1/3,1/2,1) (1/4,1/3,1/2) (1/3,1/2,1) (1,1,2) (1,1,2)

(2,3,4) (1,1,2) (1,1,2) (1/3,1/2,1) (1,1,2) (1,2,3) (1,1,2) (1,2,3) (2,3,4) (2,3,4)

(5,6,7) (1,2,3) (1,2,3) (1,1,2) (1,2,3) (3,4,5) (1,2,3) (3,4,5) (5,6,7) (5,6,7)

(2,3,4) (1,1,2) (1,1,2) (1/3,1/2,1) (1,1,2) (1,2,3) (1,1,2) (1,2,3) (2,3,4) (2,3,4)

(1,2,3) (1/3,1/2,1) (1/3,1/2,1) (1/5,1/4,1/3) (1/3,1/2,1) (1,1,2) (1/3,1/2,1) (1,1,2) (1,2,3) (1,2,3)

(2,3,4) (1,1,2) (1,1,2) (1/3,1/2,1) (1,1,2) (1,2,3) (1,1,2) (1,2,3) (2,3,4) (2,3,4)

(1,2,3) (1/3,1/2,1) (1/3,1/2,1) (1/5,1/4,1/3) (1/3,1/2,1) (1,1,2) (1/3,1/2,1) (1,1,2) (1,2,3) (1,2,3)

(1,1,2) (1/4,1/3,1/2) (1/4,1/3,1/2) (1/7,1/6,1/5) (1/4,1/3,1/2) (1/3,1/2,1) (1/4,1/3,1/2) (1/3,1/2,1) (1,1,2) (1,1,2)

Table 4. Pairwise processes comparison matrix in domain D1.

P2,1 P2,2 P2,3 P2,4 P2,5 P2,6 P2,7

P2,1

P2,2

P2,3

P2,4

P2,5

P2,6

P2,7

(1,1,2) (1,1,2) (1/9,1/9,1/8) (1/5,1/4,1/3) (1/9,1/9,1/8) (1/9,1/9,1/8) (1/3,1/2,1)

(8,9,9) (8,9,9) (1,1,2) (8,9,9) (1,1,2) (1,1,2) (8,9,9)

(3,4,5) (3,4,5) (1/9,1/9,1/8) (1,1,2) (1/9,1/9,1/8) (1/9,1/9,1/8) (1,2,3)

(8,9,9) (8,9,9) (1,1,2) (8,9,9) (1,1,2) (1,1,2) (8,9,9)

(1,2,3) (1,2,3) (1/9,1/9,1/8) (1/3,1/2,1) (1/9,1/9,1/8) (1/9,1/9,1/8) (1,1,2)

Table 5. Pairwise processes comparison matrix in domain D2.

P3,1 P3,2 P3,3 P3,4 P3,5 P3,6 P3,7 P3,8 P3,9 P3,10 P3,11 P3,12 P3,13

P3,1

P3,2

P3,3

P3,4

P3,5

P3,6

P3,7

P3,8

P3,9

P3,10

P3,11

P3,12

P3,13

(1,1,2)

(8,9,9)

(3,4,5)

(1,2,3)

(8,9,9)

(3,4,5)

(8,9,9)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/5,1/4,1/3)

(8,9,9)

(1,1,2)

(1/3,1/2,1)

(8,9,9)

(1,1,2)

(8,9,9)

(1/9,1/9,1/8)

(1/3,1/2,1)

(8,9,9)

(1,2,3)

(1,1,2)

(8,9,9)

(1,2,3)

(8,9,9)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/5,1/4,1/3)

(8,9,9)

(1,1,2)

(1/3,1/2,1)

(8,9,9)

(1,1,2)

(8,9,9)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

(1/9,1/9,1/8)

(1,1,2)

Table 6. Pairwise processes comparison matrix in domain D3.

P4,1 P4,2 P4,3 P4,4

P4,1

P4,2

P4,3

P4,4

(1,1,2) (1/9,1/9,1/8) (1,2,3) (1,2,3)

(8,9,9) (1,1,2) (8,9,9) (8,9,9)

(1/3,1/2,1) (1/9,1/9,1/8) (1,1,2) (1,1,2)

Table 7. Pairwise processes comparison matrix in domain D4.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

270 Cobo, A., Vanti, A. A., Rocha, R.

4.4. Consistency analysis of comparison judgments Seeking to contrast the consistency of comparative judgments between the criteria made by COBIT, the AHP model proposes the calculation of a ratio of consistency. In this sense Saaty (1980) recommends a radius of consistency of 0.1 or lower so that pairwise comparisons undertaken by the decision maker can be considered as acceptable. In the case proposed, we used the crisp numbers associated to the fuzzy triangular comparison values to obtain the radius of consistency of the crisp pairwise matrices; the values obtained were as follows: â&#x20AC;˘

For first level comparisons we obtained a radius of consistency of 0.02.

â&#x20AC;˘

For second level comparisons we obtained radii of consistency of 0.00, 0.04, 0.02, and 0.02 respectively. All ratios of consistency were, therefore, perfectly admissible.

4.5. Calculation of the relative weights of each criterion Once the pairwise comparison has been performed, the FAHP methodology allows us to calculate weights for each criterion which will influence their importance in achieving the final goal. We applied the methodology proposed by Chang (1996) in order to calculate these weights in Local (L) and Global (G), considering the strategic alignment focus area. The computation of these weights was performed using an ad hoc implemented application in Java programming language. Table 8 shows the final values obtained for the weights in the two levels of the hierarchy.

Plan and Organize PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10 Acquire and Implement AI1 AI2 AI3 AI4 AI5 AI6 AI7

Focus Area: Strategic Alignment L G 0,678 Deliver and Support 0,1853 0,1256 DS1 0,0599 0,0406 DS2 0,0517 0,0351 DS3 0 0 DS4 0,0425 0,0288 DS5 0,1305 0,0885 DS6 0,0318 0,0216 DS7 0,1278 0,0866 DS8 0,1853 0,1256 DS9 0,1853 0,1256 DS10 DS11 0,072 0,3006 0,0216 DS12 0,3006 0,0216 DS13 0 0 Monitor and Evaluate 0,1729 0,0124 ME1 0 0 ME2 0 0 ME3 0,226 0,0163 ME4

Table 8. Priority weights for criteria

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

L 0,2937 0 0,2283 0,2519 0 0 0,2262 0 0 0 0 0 0 0,2691 0 0,3655 0,3655

G 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0,25 0,0673 0 0,0914 0,0914

271

A Fuzzy Multicriteria Approach For IT Governance Evaluation

Table 8 prioritizes specifically strategic alignment COBIT focal area, proposed in this paper. In future works it will be applied in other focal areas as Value Delivery, Risk Management, Resource Management and Performance Measurement, using the same methodological procedure. As can be observed the model identifies only 16 key processes in making an assessment of the focal area strategic alignment. Specifically, these processes are those with non-null global weights in Table 8. As can be observed, some processes in domains closely related with strategic alignment have null weights assigned by the model. For instance, despite of PO4 is of secondary influence like other process in domain â&#x20AC;&#x153;Plan and Organizeâ&#x20AC;?, as PO3, in the mapping of IT processes to IT governance focus areas, COBIT 4.1 declares PO4 with a low importance and the weighting scheme assigns to it a null weight. In order to measure the maturity levels of the different processes defined by COBIT, it is possible to request independent evaluations by an expert group and to collect multiple crisp data sets reflecting different opinions and then form the fuzzy maturity level (FML) of a process as the triangular fuzzy number (min,mean,max) deriving from the specific data sets. To evaluate the applicability of the proposed model, we performed an assessment from the point of view of IT governance in a Brazilian company that operates in three states of Rio Grande do Sul. Individual interviews were conducted with the IT manager and supervisor, asking each of them independently conduct an assessment on the scale 0-5 the degree of maturity of each of the processes of COBIT 4.1. With the responses generated the FML values shown in Table 9 were obtained. As shown in this preliminary assessment most processes are in repeatable and defined states, especially those who have more weight in measuring the level of strategic alignment of IT in the organization according to the FAHP model developed. None of the processes can be considered optimized. Process PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10

FML (3; 3,5; 4) (2; 2; 2) (2; 2; 2) (2; 2,5; 3) (1; 2; 3) (3; 3; 3) (1; 1; 1) (2; 2; 2) (2; 2,5; 3) (3; 3,5; 4)

Process AI1 AI2 AI3 AI4 AI5 AI6 AI7

FML (3; 3,5; 4) (3; 3,5; 4) (2; 2,5; 3) (2; 2,5; 3) (1; 2,5; 4) (4; 4; 4) (4; 4; 4)

Process DS1 DS2 DS3 DS4 DS5 DS6 DS7 DS8 DS9 DS10 DS11 DS12 DS13

FML (3; 4; 5) (3; 3,5; 4) (2; 3; 4) (2; 3; 4) (2; 2,5; 3) (3; 3,5; 4) (1; 1; 1) (3; 4; 5) (2; 2,5; 3) (2; 3; 4) (2; 2,5; 3) (3; 3; 3) (4; 4; 4)

Process ME1 ME2 ME3 ME4

FML (3; 3,5; 4) (2; 2,5; 3) (1; 1,5; 2) (2; 2,5; 3)

Table 9. Fuzzy maturity levels (FML). The global weights calculated with the FAHP model and shown in Table 8, allow to obtain an overall strategic alignment assessment of IT governance in the studied company. The overall assessment is obtained by performing a weighted average

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

272 Cobo, A., Vanti, A. A., Rocha, R.

of the levels of maturity of each of the 16 processes with non-null weight. In the studied case the global evaluation of the “strategic alignment” focus area is (2,341; 2,711; 3,082). It means that the maturity level comprehends a larger interval of compliance in the company, when the IT processes are evaluated. This way, there is more alignment between IT and Business. Analyzing the results in the company, it could be said that the level of strategic alignment of IT in the organization is not properly managed. An improvement in several processes is required, especially in those that received a lower evaluation of maturity and have a greater impact on achieving an optimal level of alignment; in this case, the processes that have to be improve are PO8 and PO9 in the “plan and organize” domain, and processes ME3 and ME4 in the “monitor and evaluate” domain. 5.

CONCLUSIONS

This work provided a new multi-criteria approach to assess IT Governance (ITG) for strategic alignment between IT and Business using Fuzzy Analytical Hierarchy Process (FAHP) in the COBIT framework, more specifically to the maturity levels, domains and processes, thus providing a differentiated analysis of importance for each item. This research demonstrated that the combined use of multi-criteria decision methodologies and soft computing proves to be particularly suitable for the evaluation of IT strategic alignment. The model generated improvements the classic focus of the individual and isolated of process that are audited. This way, it is possible to better consider the choices in intervals that correspond to more qualitative, subjective or fuzzy analysis. This model has been applied to a big retail enterprise, located in the south of Brazil, through individual interviews to generate Fuzzy maturity levels (FML). For future studies, the authors will direct efforts to increase the model in other focal areas of COBIT as CMMI© (Capability Maturity Model Integration) which means the best practices to the development and maintenance activities for new projects. Furthermore, in future works we will try to adapt the multi-criteria model to the latest version of COBIT, version 5.0 includes a new domain and reorganizes some of the processes. With the corresponding adjustments to the hierarchy of processes, the methodology can be generalized to integrate the latest changes of framework COBIT in the evaluation model. We also propose as future work the integration of an Analytical Network Process (ANP) model to define more complex interrelationships between processes to evaluate.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

A Fuzzy Multicriteria Approach For IT Governance Evaluation

273

REFERENCES Balli, S. & Korukoglu, S. (2009). Operating System Selection Using Fuzzy AHP and TOPSIS. Mathematical and Computational Applications, 14(2), 119-130. Benítez, J., Delgado-Galván, X., Izquierdo, J. & Pérez-García, R. (2012). An approach to AHP decision in a dynamic context. Decision Support Systems, 53, 499-506. Cadbury Committee (2011). Report of the Committee on the Financial Aspects of Corporate Governance. London, 1992. Retrieved March 8, 2014, from http://www.jbs.cam.ac.uk/cadbury/report/index.html Chang, D. (1992). Extent analysis and synthetic decision. Optimization techniques and applications, World Scientific, Singapore, 1, 352. Chang, D. (1996). Applications of the extent analysis method on fuzzy AHP. European Journal of Operational Research, 95, 649–655. Chen, P.C. (2009). A Fuzzy Multiple Criteria Decision Making Model in Employee Recruitment. International Journal of Computer Science and Network Security, 9(7), 113-117. Chen, L. (2010). Business–IT alignment maturity of companies in China. Information & Management, 47, 9–16. De Haes, S., Van Grembergen, W. & Debreceny, R. (2013). COBIT 5 and Enterprise Governance of Information Technology: Building Blocks and Research Opportunities. Journal of Information Systems, 27(1), 307. Hirschheim, R., & Sabherwal, R. (2001). Detours in the path toward strategic information systems alignment. California Management Review, 44 (1), 87-108. IBGC (2012). Código das melhores práticas de governança corporativa. 4ª ed., São Paulo, Instituto Brasileiro de Governança Corporativa (IBGC). Retrieved March 8, 2014, from http://www.ibgc.org.br/CodigoMelhoresPraticas.aspx Iranmanesh, H., Shirkouhi, S.N. & Skandari, M.R. (2008). Risk Evaluation of Information Technology Projects Based on Fuzzy Analytic Hierarchal Process. World Academy of Science, Engineering and Technology, 40, 351-357. ISACA, Information Systems Audit and Control Association (2012). COBIT Five: A Business Framework for the Governance and Management of Enterprise IT. Rolling Meadows, IL: ISACA. ITGI (2013). COBIT 4.1: Framework, Control Objectives, Management Guidelines, Maturity Models. Information Technology Governance Institute. Retrieved March 8, 2014, from http://www.isaca.org Jensen, M. & Meckling, W. (1976). Theory of the firm: managerial behavior, agency costs and capital structure. Journal of Financial Economics, 3, 305-360. Kwok, L. & Longley, D. (1999). Information security management and modeling, Information Management & Computer Security, 7(1), 30-40.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

274 Cobo, A., Vanti, A. A., Rocha, R.

Lederer, A. L. & Mendelow, A. L. (1989). Coordination of information systems plans with business plans. Journal of Management Information Systems, 6 (2), 5-19. Lederer, A. L. & Sethi, V. (1992). Root Causes of Strategic Information System Planning Implementation Problems. Journal of Management Information Systems, 9 (1), 25-45. Li, Y. & Tan, C.H. (2013). Matching business strategy and CIO characteristics: The impact on organizational performance. Journal of Business Research, 66(2), 248–259. Lien, C.T. & Chan, H.L. (2007). A Selection Model for ERP System by Applying Fuzzy AHP Approach. International Journal of the Computer, the Internet and Management, 15(3), 58-72. Markaki, O., Charilas, D. & Askounis, D. (2010). Application of Fuzzy Analytic Hierarchy Process to Evaluate the Quality of E-Government Web Sites. DESE-10 Proceedings of the 2010 Developments in E-systems Engineering, 219-224. Marnewick, C. & Labuschagne, L. (2011). An investigation into the governance of information technology projects in South Africa. International Journal of Project Management, 29, 661–670. Meixner, O. (2009). Fuzzy AHP Group Decision Analysis and its Application for the Evaluation of Energy Sources. Proceedings of the 10th International Symposium on the Analytic Hierarchy/Network Process Multi-criteria Decision Making. University of Pittsburgh, USA. Merali, Y., Papadopoulos, T. & Nadkarni, T. (2012). Information systems strategy: Past, present, future? Journal of Strategic Information Systems, 21, 125-153. Mikhailov, L. & Tsvetinov, P. (2004). Evaluation of services using a fuzzy analytic hierarchy process. Applied Soft Computing, 5(1), 23–33. Neto, J.S., Neto, A.N.F. (2013). Metamodel of the IT governance framework COBIT. Journal of Information Systems and Technology Management, 10(3), 521-540. Preittigun, A., Chantatub, W. & Vatanasakdakul, S. (2012). A Comparison between IT Governance Research and Concepts in COBIT 5. International Journal of Research in Management & Technology, 2(6), 581-590. Saaty, T. (1980). The Analytical Hierarchy Process: Planning, Priority Setting, Resource Allocation. New York: Mc Graw-Hill. Simonson, M., Johnson, P. & Ekstedt, M. (2010). The Effect of IT Maturity on IT Governance Performance. Information Systems Management, 27, 10-24. Taylor, F. & Fitzgerald, T. (2007). Clarifying the Roles of Information Security: 13 Questions the CEO, CIO, and CISO Must Ask Each Other. Information Systems Security, 16(5), 257-263. Tiwana, A. & Konsynski, B. (2010). Complementarities between organizational IT architecture and governance structure. Information Systems Research, 21(2), 288–304. Van Laarhoven, P. & Pedrycz, W. (1983). A Fuzzy Extension of Saaty’s Priority Theory. Fuzzy Sets and Systems, 11, 229-241.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

A Fuzzy Multicriteria Approach For IT Governance Evaluation

275

Van Grembergen, W. V. & Haes, S. D. (2004). IT Governance and Its Mechanisms, Information Systems Control Journal, 1, 27-33. Van Grembergen, W., & De Haes, S. (2009). Enterprise Governance of Information Technology: Achieving Strategic Alignment and Value. New York: Springer, 233. Wajeeh, I.A. & Muneeza, A. (2012). Strategic corporate governance for sustainable mutual development. International Journal of Law and Management, 54(3), 197-208. Weill, P. & Ross, J.W. (2004). IT Governance - How Top Performers Manage IT Decision Rights for Superior Results. Boston. Massachusetts: Harvard Business School Press. Weiss, J. & Thorogood, A. (2011). Information Technology (IT)/Business Alignment as a Strategic Weapon: A Diagnostic Tool. Engineering Management Journal, 23(2), 30-41. Zadeh, L.A. (1965). Fuzzy Sets. Information and Control, 8, 338-353. ZarviÂ´c, N., Stolze, C., Boehm, M. & Thomas, O. (2012). Dependency-based IT Governance practices in inter-organisational collaborations: A graph-driven elaboration. International Journal of Information Management, 32(6), 541â&#x20AC;&#x201C;549. Zviran, M. (1990). Relationships between Organizational and Information Systems Objectives: Some Empirical Evidence. Journal of Management Information Systems, 7(1), 66-84.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

276 Cobo, A., Vanti, A. A., Rocha, R.

ANNEX: 34 PROCESSES DEFINED IN COBIT 4.1 Planning and Organization PO1 Define a Strategic IT Plan PO2 Define the Information Architecture PO3 Determine Technological Direction PO4 Define the IT Processes, Organization and Relationships PO5 Manage the IT Investment PO6 Communication Management Aims and Direction PO7 Manage IT Human Resources PO8 Manage Quality P09 Assess and Manage IT Risks P10 Manage Projects Acquisition & Implementation AI1 Identify Automated Solutions AI2 Acquire and Maintain Application Software AI3 Acquire and Maintain Technology Infrastructure AI4 Enable Operation and Use AI5 Procure IT Resources AI6 Manage Changes AI5 Install and Accredit Solution and Changes Delivery and Support DS1 Define and Manage Service Levels DS2 Manage Third Party Services DS3 Manage Performance and Capacity DS4 Ensure Continuous Service DS5 Ensure Systems Security DS6 Indentify and Allocate Costs DS7 Educate and Train Users DS8 Manage Service Desk and Incidents DS9 Manage the Configuration DS10 Manage Problems DS11 Manage Data DS12 Manage the Physical Environment DS13 Manage Operations Monitor and Evaluate ME1 Monitor and Evaluate IT Performance ME2 Monitor and Evaluate Internal Control ME3 Ensure Compliance with External Requirements ME4 Provide IT Governance

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.257-276

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 277-296 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200004

USING ANALYTIC NETWORK FOR SELECTION OF ENTERPRISE RESOURCE PLANNING SYSTEMS (ERP) ALIGNED TO BUSINESS STRATEGY Alberto de Medeiros Jr. Gilberto Perez Sérgio Lex Universidade Presbiteriana Mackenzie, São Paulo, SP, Brasil _____________________________________________________________________ ABSTRACT The choice of an Enterprise Resource Planning (ERP) must be made judiciously by the high costs involved in the acquisition of such systems. Managers in areas such as accounting, financial and information technology need support and tools that help in selecting an appropriate ERP for their business. With this article, we present a study aimed at investigating the possibility of a Decision Support System (DSS) to be used for this selection interrelating evaluation criteria, which could allow to contemplate the strategic alignment between Business and Information Technology. From the literature review 28 factors related to the selection of software packages, with special emphasis _____________________________________________________________________________________________ Manuscript first received/Recebido em: 06/05/2012 Manuscript accepted/Aprovado em: 30/04/2014 Address for correspondence / Endereço para correspondência Alberto de Medeiros Jr., Universidade Presbiteriana Mackenzie, Brasil, Doutor (2007) e Mestre (2002) em Engenharia de Produção pela Escola Politécnica da Universidade de São Paulo, graduado em Comunicação Social (1976) pela Fundação Armando Álvares Penteado (FAAP). Professor da Graduação em Administração das disciplinas Sistemas de Informação no Ambiente Corporativo e Metodologia Científica e Coordenador de Pesquisa e Extensão do Centro de Ciências Sociais e Aplicadas da Universidade Presbiteriana Mackenzie. Pesquisador do NEPEI - Núcleo de Estudos e Pesquisa em Estratégia, Empreendedorismo, Inovação e Informação da Universidade Presbiteriana Mackenzie. Foi Diretor Geral e Sócio-gerente da Intex Brasil Ltda. (1998 a 2007), subsidiária da Intex Consulting GmbH de Wuppertal, Alemanha, produtora ERP para indústrias têxteis. E-mail: alberto.medeiros@mackenzie.br Gilberto Perez Universidade Presbiteriana Mackenzie, Brasil, Doutor (2007) e Mestre (2003) em Administração pela Universidade de São Paulo, é graduado em Engenharia Eletrônica e Telecomunicações pela Universidade São Judas Tadeu (1982/83) e em Física pela Universidade de São Paulo (2001). Possui Pós-Graduação em Capacitação Gerencial pela FEA/FIA - USP (1999); MBA em Conhecimento, Tecnologia e Inovação pela FEA/FIA - USP (2001). Professor do Programa de Pós-Graduação em Administração (PPGA) na disciplina Inteligência Competitiva do Centro de Ciências Sociais e Aplicadas da Universidade Presbiteriana Mackenzie. Pesquisador do NEPEI - Núcleo de Estudos e Pesquisa em Estratégia, Empreendedorismo, Inovação e Informação da Universidade Presbiteriana Mackenzie. Email: gperez@usp.br Sérgio Lex Universidade Presbiteriana Mackenzie, Brasil, Doutor (2008) e Mestre (1999) em Administração de Empresas pela Universidade Presbiteriana Mackenzie, é graduado em Engenharia Mecânica pela Universidade Presbiteriana Mackenzie (1970). Professor titular e Decano de Extensão e Educação Continuada da Universidade Presbiteriana Mackenzie. Pesquisador do NEPEI - Núcleo de Estudos e Pesquisa em Estratégia, Empreendedorismo, Inovação e Informação o da Universidade Presbiteriana Mackenzie. E-mail: sergio.lex@mackenzie.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

278 Medeiros Jr., A. de, Perez, G., Lex, S.

on ERP were identified. For the research procedures, the qualitative ones were adopted, in that the 18 factors considered relevant to a good selection of ERP were classified with the Delphi technique and used as input in a DSS: Analytical Network Process (ANP), applied as a Case Study in a small business that hired the ERP. The results showed that the ANP was efficient in interrelated criteria and evaluated the strategic alignment between Business and Information Technology. Keywords: Selection of Information System; Enterprise Resource Planning â&#x20AC;&#x201C; ERP; Analytic Network Process â&#x20AC;&#x201C; ANP

1. INTRODUCTION Enterprise Resource Planning (ERP) systems comprise a set of applications that enable companies to automate and integrate a substantial portion of their processes, involving finances, controls, logistics, purchasing, manufacturing, sales and human resources, making it possible to share data and to standardize business processes, in addition to producing and using information in real time (Turban, Leidner, Mclean, & Wetherbe, 2010; Laudon & Laudon, 2011; Oâ&#x20AC;&#x2122;Brien & Marakas, 2013). Because ERP systems have a high cost of licensing and implementation, and a high degree of uncertainty regarding results, acquisition thereof generates insecurity among decision-makers at the time of selecting a system for the organizations, and is therefore a complex and non-structured decision (Turban et al., 2010). Cases of failure in the implementation of integrated management systems at large corporations are frequent and widely known by the Information Technology (IT) community. According to Ganly (2011), studies by Gartner Research show that 20% to 35% of the implementations fail and that the outcome for up to 80% of the projects are questionable, because they overrun the proposed deadlines and/or budgets. Better developed and friendlier solutions in terms of parameterization, better training of consultants, and greater awareness by companies regarding Critical Success Factors (CSF) in the implementation of systems have all contributed to reducing such failures. From the high number of questionable projects, however, one can deduce that there are still many gaps to be filled, evidencing the need for better management, not only for implementation planning, but also for selection when acquiring the system. Ganly (2011) considers the six major pitfalls in the implementation of an ERP system: 1) inappropriate or insufficient scope of ERP; 2) lack of commitment by managers; 3) insufficient or inappropriate budget; 4) inappropriate administration and training; 5) inexperienced management and project team; and 6) extensive modifications. With the exception of the second item, the others are the result (albeit indirectly) of poorly developed projects and poorly routed acquisition. Accordingly, with this article we sought to answer the following research problem: How to formulate a decision-making procedure for the selection of an Enterprise Resource Planning system aligned to the business that allows interrelation of evaluation criteria?

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

279

The relevance is this topic is evident, since there is still a considerable number of companies that have not yet used ERP software packages to integrate the data and processes of their departments in a single system. Studies by the Center for Information Technology and Communication Studies (CETIC, 2011) – an organ of the Brazilian Internet Steering Committee – show that the proportion of these companies represented, on average, 61% of all enterprises in 2010 and that, even at companies with over 250 employees, 27% do not use ERP (table 1). Table 1 - Proportion of enterprises that use ERP software packages to integrate data and processes of their departments in a single system. Porte

Yes

DK/NR

Total 10–49 50–249 250 or more

35 31 51 72

61 65 46 27

4 4 3 1

Source: Adapted from CETIC (2011)

The overall aim of this article is to propose a model supported by a Decision Support System (DSS), which allows companies – whenever there is an interest in acquiring an Enterprise Resource Planning system – to analyze which of the available offers will be most suited to their business strategies. 2. THEORETICAL REFERENCE 2.1 Investments in Information Technology and Information Systems According to Perez (2007), organizations in the age of information and knowledge have been investing higher amounts increasingly in Information Technology (IT) and Information Systems (IS), because IT and ISs have gradually become a significant component in practically everything that companies do (Turban et al., 2010). Authors such as Lunardi, Becker & Maçada (2003), and Perez (2007) report that some sectors have invested heavily in IS and IT, and indicate competition and rivalry as the primary factors that justify this practice. Another factor related to investments in IS and IT is the pursuit of competitiveness and above-average returns. Investments in IS and IT to obtain better security conditions have increased substantially in recent years, particularly in the financial sector. Applegate, Austin & McFarlan (2003) consider that investments in IS and IT can contribute to knowledge management; the results obtained in this case are: improvement in the performance of the knowledge of company employees and organizational learning. 2.2 Enterprise Resource Management (ERP) Information Systems perform three major roles in all types of organizations (O’Brien & Marakas, 2013): 1) Support operations; 2) Support competitive strategies; and 3) Support decisions.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

280 Medeiros Jr., A. de, Perez, G., Lex, S.

The support of operations is obtained mainly by transactional systems. The support of competitive strategies can take place through innovative uses of information systems. Decision support is ultimately made by managerial or executive information systems, by Data Warehouse and Data Mining resources, or by DSS (O’Brien & Marakas, 2013). Highlighted among transactional systems is ERP, which can be conceptualized as being a comprehensive and integrated software package that enables the standardization and automation of business processes by using a unified database and real-time transactions. Implementation of ERP at companies eliminates the “islands of automation” (McFarlan & McKenney, 1982) that existed prior to this. 2.2.1 Reasons for Adopting ERP The option of adopting an ERP system – despite the fact that it is a common desire of top executives of all companies – is always a complex decision. However, one can clearly see that there are often expectations beyond what the applications typically offer. Colângelo (2001) considers that there are three classes of reasons that lead a company to implement an ERP system:  Business: aims to increase profitability and strengthen the organization’s competitive position.  Legislation: aims to meet the legal requirements that the company must comply with and not fulfilled by legacy systems (those in use at the company).  Technology: aims to meet the necessary changes resulting from economic obsolescence of technologies in use or the requirements of business partners. However, there are several common minimum expectations (Koch, 1999; Colângelo, 2001), the most mention-worthy of which are (among others): a) Managerial or executive information; b) Integrated financial information; c) Reduction of inventory; and d) Lack of integration among legacy systems. 2.2.2 Critical Success Factors in Implementing ERP Critical Success Factors (CSFs) represent a mechanism for identifying information needs by the managers of organizations. For Rockart (1979, p. 85), “CSFs are, for any business, the limited number of areas in which results, if they are satisfactory, will ensure successful competitive performance for the organization.” These are the few key areas where “things have to go right” for the business to succeed. Esteves-Sousa and Pastor-Collado (2000) compiled the CSFs found in the literature, in ten studies based on the analysis of reports on deployment of Enterprise Resource Planning systems. They determined the similarities and common patterns among them, and proposed the unification thereof according to the model shown in Figure 1.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

Level Perspective

281

FCS

Technological

Organizational

Continuous support from company management. Management of effective organizational changes. Management of a good project scope. Appropriate composition of the project team. Strategic Comprehensive process of business reengineering. Role of the project sponsor. Participation and involvement of the users. Reliability among partners. Dedication of the work group and consultants. Strong internal and external communication. Design and scheduling of formal implementation. Appropriate training program. Tactical Preventive troubleshooting. Appropriate use of consultants. Authorized decision makers. Appropriate strategy of implementation. Avoiding customizations. Strategic Appropriate system version. Appropriate configuration of the software. Tactical Knowledge of legacy systems. Figure 1. Relevance of the CSFs per perspective. Source: Esteves-Sousa & Pastor-Collado (2000)

2.2.3 Selecting an ERP platform Selecting ERP suppliers is a complex task that requires an effort on the part of the contracting enterprise, under penalty of failure and generation of an environment of animosity in the customer-supplier relationship, if the selection does not abide by the pre-established criteria (Perez & Zwicker, 2005). Thus, more than merely seeking suppliers, companies must seek lasting partnerships with their suppliers. Perez & Zwicker (2005) emphasize that technological training, the prospect of a lasting relationship, ethical stance, excellence of the services provided, and expandability of future solution offerings are, among others, relevant factors and should be taken into account when choosing a supplier. Due to its characteristics, the selection of an application such as ERP must follow criteria previously defined by managers/decision makers. Mendes & EscrivĂŁo Filho (2007) present a roadmap considered as ideal for acquiring ERP systems and that allows an assessment of the adaptations and impacts on the changes of the organization. The aforementioned study, however, offers no guidance on the choice of system or supplier. The selection process usually starts by issuing a Request for Proposals (RFP), which is an invitation to potential suppliers to submit their offers of specific products or services to the organization. This document is a questionnaire that involves the organizationâ&#x20AC;&#x2122;s requirements, not only regarding price, but also including more in-depth information about the supplier of the product or service, such as: financial data, technical

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

282 Medeiros Jr., A. de, Perez, G., Lex, S.

competence, forecast for implementation, business references and, primarily, details about the product in order to verify its degree of fulfillment of the purchasing company’s business needs. The requirements vary from company to company and often reach hundreds of items to be analyzed. 2.2.3.1 ERP Selection Criteria

1. Technical criteria 2. Functionality 3. Supplier’s references 4. Cost 5. “Implementability” 6. Ease of customization 7. Cross modular integration 8. Consultancy of selection and implementation 9. Strategic fit 10. Risks 11. Flexibility 12. Analysis of benefits 13. Service and support 14. System reliability 15. Best fit with the organizational structure 16. Fit with the system of the parent company and/or partners 17. Method for deploying software 18. Supplier’s knowledge domain 19. Supplier’s position on the market 20. Compatibility with other systems 21. Supplier’s vision 22. Administrative changes 23. Implementation Time 24. “Scalability” to allow for growth 25. Technology upgrades 26. Ease of use 27. Security 28. Localization Figure 2. Selection criteria identified Source: Based on the literature researched.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

X X X X X

X X X X X X X X

X X X

X X X X

X X

X X X X X

X X X X

Corrêa (1995)

Fitzgerald (1998)

Illa et al (2000)

Kumar et al (2002)

Baki et al (2005)

Teltumbde (2000)

Figure 2 shows the results of research in the existing literature on the criteria most often considered in the evaluations. These criteria can be used in DSS, with the purpose of evaluating which ERP platform would be the most appropriate to the purchasing company.

X X X X X X X X X X X X X

X X X X X X X X

X X

www.jistem.fea.usp.br

X X X X X

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

283

2.4 Strategic Alignment of IT with the Corporate Strategy According to Chan & Huff (1993), strategic alignment is the integration of IT with an organizationâ&#x20AC;&#x2122;s key strategies and core competencies. Henderson & Venkatraman (1993) developed a model to analyze the strategic alignment of IT with corporate strategy, which presents four blocks that represent the four domains of the strategic choice (Figure 3).

Figure 3. Strategic alignment model Source: Adapted from Henderson & Venkatraman (1993)

The pair of blocks on the left focuses on Business, while the pair on the right focuses on Information Technology. From another standpoint, the upper pair (of the external domains) represents Strategy, while the lower pair (of the internal domains) represents Infrastructure. The interrelationship is such that any strategic decision involves decisions in one or more domains. One domain guides or impacts activities in one or more of the other domains. The business strategy, for example, impacts the organizational infrastructure, and the IT strategy impacts the IT infrastructure. This model allows organizations to analyze in which domains they are strong or weak, and the effects of a decision on one domain on the others. Although strategic alignment is usually done in the external domain, i.e., in the relationship between business strategies and IT strategies, this does not mean that the internal domain (relationship between organizational infrastructure and IT infrastructure) is secondary or any less important. Consider two types of integration between the domains of business and IT. One is strategic integration, which links business strategy to IT strategy, reflecting the external domains. Another is the functional integration, reflecting the internal domains, which link the organizational infrastructure to the IT infrastructure.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

284 Medeiros Jr., A. de, Perez, G., Lex, S.

Each domain also contains three sets of choice, as identified in Figure 3 and described in Figure 4. Henderson & Venkatraman (1993) consider that strategic alignment is a simultaneous result of both a strategic fit and a functional integration. The strategic fit is obtained when decisions are made, which properly position the company on the market, when in the business domain, or technologies at the company, when in the IT domain. A good strategic fit allows the structure, processes and people skills to be capitalized in the implementation of the organizational strategy, in the Business domain or of technology in the IT domain (Luftman, Lewis, & Oldach 1993; Curtin, 1996).

S t r a t e g y

Business

Information Technology

Scope of the business: The decisions and choices that define the companyâ&#x20AC;&#x2122;s field of competition, such as: products, niche, customers, etc. Includes the competitive forces proposed by Porter (1989). The typical questions are: what is our business? What are our products, services and target markets?

Scope of technology: Specifies the critical technologies responsible for the organizationâ&#x20AC;&#x2122;s success, such as knowledge base systems, robotics, multimedia, etc. The typical question is: what technologies not only support but also create strategic business opportunities?

Distinctive competencies: These are the areas that determine how companies will compete in delivering their products or services. It is the companyâ&#x20AC;&#x2122;s ability to differentiate its products or services from the competition. The typical question is: what we should focus on doing better in order to distinguish ourselves from our competitors?

Systemic competencies: Important characteristics of IT that are critical in the creation or extension of business strategies, such as connectivity, accessibility, reliability, etc. The typical question is: what characteristics of IT create business advantages?

Business governance: These are decisions relating to possible partnerships or outsourcing, in order to obtain advantages of scale for catering to a particular market. The typical question is: what relationships with external business and/or joint ventures do we depend on?

IT Governance: Defining the ownership of the technology (end user, management committee, etc.) or the possibility of technological alliances such as partnerships, outsourcing, or both, or make-or-buy decisions. The typical question is: what external relationships such as outsourcing, make-or-buy decisions, etc., do we depend on?

Figure 4. Strategic alignment model Source: Adapted from Luftman, Lewis & Oldach (1993) and Curtin (1996)

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

I n f r a s t r u c t u r e

285

Business

Information Technology

Administrative infrastructure: These are choices that establish the framework of management and work processes that a company will operate. It is the framework of rules, responsibilities, and powers of the company. Typical questions are: what is our organizational structure? Who reports to whom?

Architectures: These are the choices, priorities or policies that synthesize the applications, data, hardware and software into a cohesive platform. The typical question is: what are our options of platforms, hardware, software, network settings and data architecture?

Processes: There are choices that determine how the basic business functions will operate or flow. Define how restructured or integrated the workflows will be in order to improve the efficiency and effectiveness of companies. The typical question is: what are our key business processes?

Processes: This is the configuration as to the practices of application development and management control of the systems. The typical question is: what are the IT processes: development, maintenance, system operations, database administration?

Skills: These are choices where human resources involved in the strategy and the need to hire outsourced services must be clearly defined. The typical question is: what human resources do we have (or do we need) to keep up with our specific competitors?

Skills: These are the experiences, competencies, commitments, values and standards that govern the tasks with the aim of production and delivery of IT products and/or services. The typical question is: what are the skills that our IT managers, and supporting staff need to keep the architecture and execute the processes?

Figure 4. Strategic alignment model (cont.) Source: Adapted from Luftman, Lewis & Oldach (1993) and Curtin (1996)

2.5 Decision Support Systems According to Shimizu (2010), there are several methods that enable decision support, among which are the following:  Utility theory assumes that a decision-maker always seeks the solution that generates the greatest satisfaction or “utility” (i.e. usefulness) for them.  ELECTRE methods (from the French acronym for Elimination et Choix Traduisant La Realité): a decision-making algorithm for problems with multiple criteria, which reduces the size of the set of possible alternatives, classifying them according to the criteria of dominance of one over the other.  MACBETH method (from the acronym for “Measuring Attractiveness by a Categorical Based Evaluation Technique”): allows one to concentrate the various evaluation criteria into a single synthesis criterion by assigning weights to various criteria, respecting the opinions of the decisionmakers.  Analytic Hierarchy Process (AHP) method: the basic assumption of AHP is that a complex problem can be efficiently solved when it is broken down into several parts interconnected by a hierarchical structure, determining specific weights for each criterion – in a pairwise comparison – in order to rank the alternatives.  Analytic Network Process (ANP) Method: enables the decision maker to consider the possible existence of dependences between decision JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

286 Medeiros Jr., A. de, Perez, G., Lex, S.

factors and analyze the effect and feedback arising from these dependences. This is obtained by judgment calls and use of measurements by proportional scales. While AHP is a theory that depends on the values and judgments of individuals and groups, based on a given scenario, ANP is a generalization of AHP (Saaty, 2001), since it allows the construction of different scenarios, for complex decision problems. The main difference between AHP and ANP is that the latter is an approach that substitutes hierarchies for networks and, in both approaches to decision making, judgments are executed jointly in an organized manner to produce priorities. In ANP, interaction and feedback take place in clusters of elements with inner dependence, and between these clusters with outer dependence, better capturing the “complex effects of reciprocity in human societies, especially when risks and uncertainties occur” (Saaty, 2001). ANP, by allowing the feedback of the judgment and due to its characteristic of allowing clusters, was chosen as a research instrument of the Case Studies, since ANP is a relatively simple multi-criteria method, with an intuitive approach that can be easily accepted by managers and other decision makers (Meade & Presley, 1999). It can be applied with the use of electronic spreadsheets, such as Excel, but also allows for ease of use by applying the SuperDecisions software which can be downloaded from www.superdecisions.com. Feedback allows one to make a judgment not only regarding the importance of each alternative for each criterion, but also how these have their importance judged for each alternative. In this study, it was not intended to compare DSSs. The aim was to determine whether there is the possibility of obtaining a selection of ERP aligned with business strategies using a DSS. Therefore, we defined – albeit in a prescriptive manner – the use of ANP, since it allowed the formation of the grid according to Henderson & Venkatraman (1993) to be reproduced in the form of the clusters thereof. These clusters contain the criteria to be judged according to the dependences between decision factors and to analyze the effect and feedback arising from these dependences. This is obtained by judgments and the use of measurements by proportional scales for the distribution of influence between factors and groups of factors in the decision, enabling the allocation of resources according to the proportional scales of priorities. Judgments in ANP are pairwise using a fundamental scale (also used in AHP), where intensity 1 indicates equal importance between the two activities, 3 indicates moderate importance, 5 indicates strong importance, 7 indicates very strong importance, and 9 indicating extreme importance. Even-numbered intensities indicate intermediate importances between the odd-numbered intensities, and the reciprocal values are used if an initial activity compared to a second activity has a value equal to one of the intensity values indicated, then that second activity has the intensity value reciprocal to the first (Saaty, 2000, 2001, 2005).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

287

According to Saaty (2005, p. 47), two questions must be answered: “1) Given a criterion, which one of two elements is more dominant regarding the criterion? 2) Which of the two elements most influences a third element regarding the criterion?”

Figure 5, Abstract representation of the components of a decision in a hierarchical manner. Source: Adapted from Saaty (2005).The

simplest model has a cluster called “goal,” containing a “goal” element, a cluster called “criteria” containing the “criteria” elements, and a cluster called “alternative” containing the “alternative” elements, as shown in Figure 5. When the clusters are connected by a line, this means that the nodes (elements) are connected (Saaty, 2001). The direction of the arrow indicates the comparison relationship of the clusters. Therefore, a two-way arrow indicates that sub-criteria influence the alternatives and vice versa. The arc-shaped arrow under the “alternatives” cluster (indicating a loop) means that the elements contained therein influence one another. The elements contained in “criteria” and “sub-criteria” were considered independent. The performance of one alternative can influence that of another alternative. The performance of one criterion on a sub-criterion, however, is not dependent on the performance on other subcriterion (Salomon, 2004). 2.5.1 Fundamental components of ANP 2.5.1.1 Dependences In an ANP, the components can influence other elements in the same component. This is called inner dependence; with other components, it is called outer dependence.

Figure 6. Abstract representation of the dependences in a decision network. Source: Saaty (2005) In

Figure 6, the line between components C4 and C2 indicates outer dependences regarding common properties of the elements in C2 regarding the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

288 Medeiros Jr., A. de, Perez, G., Lex, S.

elements in C4, while the loops in components C1 and C3 indicate inner dependence in common properties of the elements in these components (Saaty, 2005). In order to avoid very complex networks, the ANP allows a cluster to be composed of a sub-network, which contains new clusters and respective nodes. 2.5.1.2 The Supermatrix The supermatrix consists of blocks of vectors of priorities for the clusters in the network, which are arranged in the supermatrix from the top down starting on the left side. Under each cluster are the nodes belonging thereto; the same thing occurs with the lines (Saaty, 2005). The comparison between all the linked clusters results in a matrix of global reach, which is used to weight the blocks of clusters in the supermatrix. While the supermatrix is not weighted by the matrix of global reach of the clusters, it is called “unweighted supermatrix”. The sum of the values in the columns will be greater than one in those that contain a node that is compared to another in an inner dependence. After the weighting is applied by the global-reach matrix of the clusters, it becomes a “weighted supermatrix.” In this case, the effect of the weighting causes the sum of each column of the supermatrix to have a value of one (Saaty, 2005). If the ANP design is created with sub-networks, each one will have its own supermatrix. 3. METHODOLOGICAL PROCEDURES 3.1 Research Techniques The research involved the combination of two techniques of a qualitative nature, by means of the Delphi technique and the Case Study strategy. With the Delphi Technique, we sought – through consensus of a group of specialists – to classify the criteria that should be considered the most appropriate in the selection of an ERP system, while the second one aimed to verify the use of the procedure in a small business. 3.1.1 Delphi technique (DT) This is an exploratory research approach (Hair, Babin, Money & Samouel, 2005). It is a qualitative analysis tool used primarily in situations in which there is a lack of historical data or technological breaks. An important aspect of the DT is to be an approach that seeks consensus among a group of specialists (Turoff & Linstone, 1975; Wright, 1986). In the first part of the survey, the revised DT was used, as proposed by Dickson & Nechis (1984), prioritizing the key points in IS, to classify the relevant criteria for evaluating the ERP. 3.1.2 Case Study Among the various qualitative research strategies, one can see that the Case Study is one of the techniques that has been used quite frequently for organizational studies, because it presents the possibility of a reflection of the results in the area of administrative sciences. This is a technique that involves an intensive and detailed study of a well- defined entity: the “case” (Coutinho & Chaves, 2002). According to Yin (2010, p. 19), the Case Study “is a preferred strategy [...] when the researcher has

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

289

little control over the events and when the focus is on contemporary phenomena inserted in some real-life context”. 4. PRESENTATION AND ANALYSIS OF THE RESULTS 4.1 Research Instrument Model In order for there to be a reduction of factors, thereby avoiding the use of all 28 criteria shown in Figure 2, which would represent a high number of pairwise comparisons and make the application of the model unfeasible, the most relevant ones were obtained from two rounds of the Delphi Technique initially applied to 41 experts on the subject matter, out of whom 27 showed valid responses. Accordingly, it was decided that the 18 most relevant assessment criteria indicated by the experts would be used in the ANP (Figure 7), which were then distributed among the clusters created in the SuperDecisions software according to the choices proposed by Henderson & Venkatraman (1993). Sub-network

Choice

Criterion Flexibility

Scope of IT

Functionality

System reliability Cost

Ease of customization

“Implementability” Systemic competencies IT Strategy Supplier references Security

Implementation time

IT Governance

Compatibility with the system of the parent company and/or partners Compatibility with other systems

Denotes the system’s ability to support the business needs during its lifetime. It has three principal aspects: 1) which functional areas the product can cover; 2) how flexible the product is regarding adaptability and openness; and 3) specific characteristics of some ERP platforms. The system should incorporate the best business practices of each area, as well as the latest trends in IT. This reliability should be verified with the users. The price of an ERP system is usually very high. The Total Cost of Ownership (TCO) of the system must include the price of licenses, maintenance and upgrades, supplementary software, hardware, network, consultancy, training, implementation team, as well as other costs. Even if one wishes to avoid customization, most companies need it, since there is a need to adapt a generic solution to the company’s specific needs. Customization can be done in-house, or by specialized consulting firms, or developed by the system producer itself, which is then incorporated into the main application. Some projects are more difficult to implement than others. Some may never be able to effectively move ahead. The implementation of ERP systems also frequently demands redesign of the company’s processes, not only so that imperfect processes are not automated, but also so that the processes become adapted to the best use of the system. The supplier’s sales references, reputation, and international scope, with a track record of successful projects, should be considered as important criteria for the selection process. Aspects related to data security and integrity should be considered when purchasing a system. Resources such as allocation and management of access passwords, protection against outside attackers, data backup and recovery support routines, and resources to deal with power outages during execution of routines are most outstanding. Implementation of an ERP system is highly costly and complex in largescale projects. It is strongly correlated with the implementation strategy, and can also occur through changes in scope. Specific solutions can also reduce implementation time. Compatibility with the system of the parent company and/or partners can affect the decision-making process of some companies. This compatibility can also affect the success of the project.

No application can do everything a company needs. There is no such thing as absolute fulfillment. The selected solution must have resources for integration with the company's legacy systems and other specialized software the company uses to cover all its needs.

Figure 7. Criteria for evaluating ERP attributed to the choices in the strategic alignment model in the sub-networks subordinate to Information Technology

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

290 Medeiros Jr., A. de, Perez, G., Lex, S.

Technical criteria

IT Architectures

IT Infrastructure

IT Processes

IT skills

Scalability to allow for growth

Appropriate configuration of the software Cross modular integration

Consulting services in selection and implementation Supplierâ&#x20AC;&#x2122;s knowledge domain Service and support

ERP is an IT application and its technological dimensions are important in environments of rapid technological changes, therefore, it must be up-todate in IT trends, particularly as related to database management systems, client-server environment, hardware, operating system, etc. Scalability allows ERP to grow gradually, according to the gradual growth of users at the company, allowing for increase in the capacity of hardware and software platforms in the same proportion, avoiding sharp discontinuities from complete changes of platform. Involves adapting the generic functionalities of a package to the particular needs of an organization. The real benefit of an ERP system is integration. And this integration should be complete among the modules. If this does not exist, aside from increasing the implementation cost, it may affect the efficiency of the system, which could decrease. The consultants should have experience in the specifics of the company, with comprehensive knowledge of the modules, and be able to determine which system resources will best meet the companyâ&#x20AC;&#x2122;s needs. Itâ&#x20AC;&#x2122;s important that the system developer has knowledge of the business segment. If the enterprise is a manufacturing company, it needs to find a software supplier specialized in its area of business. A complete installation of ERP can reach an investment many times greater than the cost of a software package. Therefore, the associated service and the support become crucial factors for the success of the business between the user and the supplier of a system.

Figure 7. Criteria for evaluating ERP attributed to the choices in the strategic alignment model in the subnetworks subordinate to Information Technology (Cont.)

Figure 8 shows the results, considering that there is dependence between and within clusters, represented by the arrows and the arcs.

a. Initial network Sub-networks

b. Business Strategy under the Business cluster

d. Business Infrastructure under the Business cluster

c. IT Strategy under the IT cluster

e. IT Infrastructure under the IT cluster

Figure 8. Graphical representation of the network and sub-networks in the choice of ERP aligned with the corporate strategy used as a research tool in the Case Study. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

291

The model was reproduced in the SuperDecisions software whose graphic resources facilitate the understanding and control of the decision maker. A research protocol was prepared with all the possibilities of pairwise judgments and applied to a Case Study, with the manager of a company who had participated in deciding the choice of its ERP platform, from among three options (ERP #1 ERP #2, ERP #3) offered by the market. We chose a small business that produces accessories for the automotive industry. The Case Study aimed to identify: 1) whether the ANP is suitable as a tool for analyzing the selection of ERP; 2) whether the use of the ANP built as the strategic alignment model (Henderson and Venkatraman’s model) allows the assessment to consider the respective importance of Business and IT assigned to the evaluation. 4.2 Results obtained At the chosen company, the impact of the judgment between Business and IT was evaluated with the ANP, in order to verify that it would reflect the need for alignment between these two areas. Several simulations were carried out, and the impact of strategic alignment between Business and Information Technology was confirmed, as contemplated in the model developed by Henderson and Venkatraman (1991). Graph in Figure 9 shows the simulation of the impact of the importance given by the company’s decision maker, where one can see that the best rated ERP after evaluation in the ANP (ERP #3) always has a value equal to 1, since this is the ideal result. The simulation shows that if the decision maker considers that IT is more important than the Business, then the possible decision for ERP #1 comes rather close ERP #3.

Figure 9. Ideal result (where the ERP with the best result – ERP #3 – has a value equal to 1), demonstrating that if the decision maker considers a higher intensity in IT, ERP #1 approaches ERP #3.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

292 Medeiros Jr., A. de, Perez, G., Lex, S.

Sensitivity analyses were conducted, considering the results of the dependent variables (the ERPs) for the independent variables (the clusters) and also reflected this for the intensities of the judgment on the importance of the Business relative to the importance of IT within the organization. The sensitivity analysis works with the “what if...” type of hypothetical scenarios, where the value of only one variable is changed repeatedly, and the changes occurring in the other variables are observed. This is done by repeated changes in only one variable at a time. This analysis enables an understanding of the impact of this variable on the others (O’Brien & Marakas, 2013). Saaty (2000, p. 112) considers that “a useful concern in any theory based on measurement is to hypothetically make both small and large perturbations in the measurements and note their effects on the outcome”. Figure 10 graphically shows these sensitivities, extracted from the SuperDecisions software. One can see that despite the fact that ERP #3 showed a growing trend in direct relation to the experiments in Business Strategy, this does not occur in IT Strategy, in which – starting at grade 0.5 of the experiments – ERP #1 is classified in the first position. In IT Infrastructure, ERP #1 has advantage over ERP #3 when IT is evaluated as extremely important compared to the Business, however this position is inverted starting from grade 0.3 in the experiments. This demonstrates that the model is sensitive, because according to Turban et al. (2010) a model is sensitive when small changes in conditions determine different solutions.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

293

1. Independent variable: Business Strategy

2. Independent variable: IT Strategy

3. Independent variable: Business Infrastructure

4. Independent variable: IT Infrastructure

a. Extreme importance of Business in relation to IT

b. Business and IT have the same importance

Figure 10. Sensitivity analysis

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

c. Extreme importance of IT in relation to the Business

294 Medeiros Jr., A. de, Perez, G., Lex, S.

5. CONCLUSION The high cost and failed or questionable implementations have made the selection of an ERP system a challenge to business organizations. A recurring difficulty for managers is how to interrelate the various selection criteria to be considered. As a result, we developed this study with the aim of answering the following research problem: How to formulate a decision-making procedure for the selection of an Enterprise Resource Planning system aligned to the business that allows interrelation of evaluation criteria? To carry out this study, the various criteria selected based on the literature were subsequently submitted to experts, who classified them according to their degree of relevance by using the Delphi Technique. Using the criteria identified by the research may be a good start, but each company has its peculiarities, which should be researched together with the master-users. However, this procedure does not result in a sine qua non. The usefulness and applicability of the results obtained in the research reside precisely in the fact that specialists on the subject of ERP participated in it, who were able to transparently classify the criteria identified in the research, based on their knowledge and experiences acquired over time. As a result, companies can use them as a decision support tools in the acquisition of information technologies and systems, as is the case with ERP. The sensitivity analysis is shown to be very useful to verify the robustness of the model, demonstrating that – depending on the intensities assigned to the clusters used as independent variables – an effective change of classification among the systems analyzed can occur. Saaty (2000, p. 112) considers that “a useful concern in any theory based on measurement is to hypothetically make both small and large perturbations in the measurements and note their effects on the outcome”. However, one should consider that the system acquired is not always shown to be aligned with the corporate strategy. Aligning strategically does not necessarily mean considering equal importance between Business and Technology. A decision maker of a company geared toward IT may consider IT Strategy more relevant than Business Strategy. The Case Study presented in this paper evaluated the use of the ANP as a DSS that could include these options of each decision maker. The evidence found during the application of the models as a research tool in the Case Studies allows us to consider that the ANP – in the cases studied as well as in the case presented herein – may enable the results necessary for appropriate decision making. 5.1. Research limitations and recommendations The research was not aimed at evaluating DSS, but rather to assess whether one such system (in this case, ANP) could include the alignment between Business and Information Technology. In this research, the selection criteria were defined only for the domains of Information Technology, leaving the definition of criteria for Business domains for future research efforts. The results presented herein are typical of the company (case) studied, therefore they cannot be generalized. However, such results may

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Using Analytic Network for Selection of Enterprise Resource Planning (ERP) Aligned to Business Strategy

295

have value for companies and managers going through the stage of procurement/selection of Enterprise Resource Planning systems. Therefore we recommended that this research be continued, using a larger number of cases, with (or even by means of) the use of the criteria for Business domains. Another possibility would be to develop quantitative studies that seek to measure the validity of the evaluation criteria rated by the experts.

REFERENCES Applegate, L. M., Austin R. D. & McFarlan, F. W. (2003) Corporate Information Strategy and Management – The challenges of managing in a network economy. McGraw-Hill. Baki, B. & Çakar, K. (2005) Determining the ERP package-selecting criteria: the case of Turkish manufacturing companies. Business Process Management Journal. 11, pp. 75-86. Cetic – Centro de Estudos sobre as Tecnologias de Informação e de Comunicação (2011) Pesquisa sobre o uso das tecnologias da informação e da comunicação no Brasil 2010. São Paulo: Comitê Gestor da Internet no Brasil, Chan, Y.E. & Huff, S.L. (1993) Strategic information systems alignment. Business Quarterly, 58, pp. 51- 55. Colângelo, L., Filho (2001) Implantação de sistemas ERP.- Um enfoque de longo prazo. São Paulo: Atlas. Corrêa, H.L. (1995) Aspectos a considerar na seleção de uma solução ERP para médias empresas. Available in: <http://www.correa.com.br/biblioteca/artigos/A04_Computerworld_artigo_escolha_softwa re.pdf>. Accessed on: May 20, 2003. Coutinho, C.P. & Chaves, J.H. (2002) O estudo de caso na investigação em tecnologia educativa em Portugal. Revista Portuguesa de Educação, 15, pp. 221-243. Dickson, G.W. & Nechis, M. (1984) Key information system issues for the 1980’s. MIS Quartely, 8, pp. 135-159. Esteves-Sousa, J. & Pastor-Collado, J. (2000) Towards the unification of critical success factors for ERP implementations. In: 10th Annual Business Information Technology (BIT) 2000 Conference, 2000. Manchester. Fitzgerald, G. (1998) Evaluating information systems projects: a multidimensional approach. Journal of Information Technology, 13, pp. 15-27. Ganly, D. (2008, August) Address Five Key Factors for Successful ERP Implementations. Gartner Research. ID Number: G00160011 Hair, J.F., Jr, Babin, B., Money, A.H. & Samouel, P. (2005) Fundamentos de métodos de pesquisa em administração. Porto Alegre: Bookman, Henderson, J.C. & Venkatraman, N. (1993) Strategic alignment: Leveraging information technology for transforming organizations. IBM Systems Journal, 32, pp. 4-16. Illa, X.B., Franch, X. & Pastor, J.A. (2000) Formalizing ERP selection criteria. In: 10TH International Workshop on Software Specification and Design, IEEE, San Diego. Koch, C. (1999) The ABCs of ERP - Getting started with Enterprise Resource Planning. CIO Magazine, December, 22. Available in <http://teaching.fec.anu.edu.au/INFS3024/Lecture%20Notes/The%20ABCs%20of%20ER P%20-%20Enterprise%20-%20CIOb.pdf>. Accessed on: October 14, 2010

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 277-296

www.jistem.fea.usp.br

296 Medeiros Jr., A. de, Perez, G., Lex, S.

Kumar, V., Maheshwari, B. & Kumar, U. (2002) Enterprise resource planning systems adoption process: a survey of Canadian organizations. International Journal of Production Research, 40, pp. 509-523. Laudon, J.P. & Laudon, K.C. (2011) Sistemas de informações gerenciais. 9 ed. São Paulo: Pearson. Lunardi, G. L., Becker, J.L. & Maçada, A. C. G. (2003) Relacionamento entre Investimentos em Tecnologia de Informação (TI) e Desempenho Organizacional: um Estudo Cross-country envolvendo os Bancos Brasileiros, Argentinos e Chilenos. Congresso ENANPAD, 2003. 1 CD-ROM McFarlan, F.W. & McKenney J.L. (1982, Sept./Oct.) The information archipelago - Maps and bridges. Harvard Business Review. 60, pp. 109-119. Mendes, J.V. & Escrivão, E., Filho (2007, maio-ago) Atualização tecnológica em pequenas e médias empresas: proposta de roteiro para aquisição de sistemas integrados de gestão (ERP). Gestão e Produção. 14, pp. 281-293. O'Brien, J.A. & Marakas, G.M. (2013) Administração de Sistemas de Informação. Uma introdução. 15. ed., Porto Alegre: McGrawHill/Bookman. Perez, G. & Zwicker, R. (2005) Fornecedores de Serviços de Tecnologia da Informação: Estratégias de Seleção. CATI - Congresso Anual de Tecnologia de Informação, São Paulo. Perez, G. (2007) Adoção de inovações tecnológicas: um estudo sobre o uso de sistemas de informação na área de saúde. 2007. 180 p + apêndices e anexos. Tese (Doutorado) – Departamento de Administração, Faculdade de Economia, Administração e Contabilidade, Universidade de São Paulo, São Paulo. Presley, A. & Meade, L. (1999) Strategic alignment and IT investment selection using the Analytic Network Process. In: Americas Conference on Information Systems (ACIS'99), Milwaukee. Rockart, J.F. (1979, Mar./Apr) Chief executives define their own data needs. Harvard Business Review. 57, pp. 81-92. Saaty, T. L. (2000) Fundamentals of decision making and priority theory: with the Analytic Hierarchy Process. Pittsburgh: RWS Publications. Saaty, T. L. (2001) Decision making with dependence and feedback. The analytic network process. 2. ed. Pittsburgh: RWS Publications. Saaty, T. L. (2005) Theory and applications of the analytic network process. Decision making with benefits, opportunities, costs, and risks. Pittsburgh: RWS Publications. Shimizu, T. (2010) Decisão nas organizações: Introdução aos problemas de decisão encontrados nas organizações e nos sistemas de apoio à decisão. 5. ed. São Paulo: Atlas. Teltumbde, A. (2000) A framework for evaluating ERP projects. International Journal of Production Research, 38, pp.4507- 4520. Turban, E.; Leidner, D.; McLean, E., & Wetherbe, J. (2010) Tecnologia da Informação para Gestão - Transformando os Negócios na Economia Digital. Porto Alegre: Bookman,. Turoff, M. & Linstone, H.A. (1975) The Delphi Method. New York: Addison Wesley Publishing Company Inc. Wright, J.T.C. (1985, novembro) A Técnica Delphi: Uma ferramenta útil ao planejamento do Brasil? In: III Encontro Brasileiro de Planejamento Empresarial São Paulo. SPE – Sociedade Brasileira de Planejamento Empresarial, pp. 199-207. Yin, R.K. (2010) Estudo de caso: planejamento e métodos. 4. ed. Porto Alegre: Bookman.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

UTILIZAÇÃO DA REDE ANALÍTICA PARA A SELEÇÃO DE SISTEMAS INTEGRADOS DE GESTÃO (ERP) ALINHADOS À ESTRATÉGIA DE NEGÓCIO Alberto de Medeiros Jr. Gilberto Perez Sérgio Lex Universidade Presbiteriana Mackenzie, São Paulo, SP, Brasil _____________________________________________________________________ ABSTRACT A escolha de um Sistema Integrado de Gestão (Enterprise Resourse Planning - ERP) deve ser feita de forma criteriosa pelos altos custos envolvidos com a aquisição deste tipo de sistema. Gestores de áreas como contabilidade, financeira e tecnologia da informação necessitam de apoio e ferramentas que os auxiliem na seleção de um ERP adequado ao seu negócio. Com este artigo, apresenta-se uma pesquisa que visou verificar a possibilidade de um Sistema de Apoio à Decisão (SAD) ser utilizado para essa seleção, inter-relacionando critérios de avaliação, que possibilitassem contemplar o alinhamento estratégico entre o Negócio e a Tecnologia de Informação. A partir da revisão da literatura foram identificados 28 fatores relacionados à seleção de pacotes de _____________________________________________________________________________________________ Manuscript first received/Recebido em: 06/05/2012 Manuscript accepted/Aprovado em: 30/04/2014 Address for correspondence / Endereço para correspondência Alberto de Medeiros Jr., Universidade Presbiteriana Mackenzie, Brasil, Doutor (2007) e Mestre (2002) em Engenharia de Produção pela Escola Politécnica da Universidade de São Paulo, graduado em Comunicação Social (1976) pela Fundação Armando Álvares Penteado (FAAP). Professor da Graduação em Administração das disciplinas Sistemas de Informação no Ambiente Corporativo e Metodologia Científica e Coordenador de Pesquisa e Extensão do Centro de Ciências Sociais e Aplicadas da Universidade Presbiteriana Mackenzie. Pesquisador do NEPEI - Núcleo de Estudos e Pesquisa em Estratégia, Empreendedorismo, Inovação e Informação da Universidade Presbiteriana Mackenzie. Foi Diretor Geral e Sócio-gerente da Intex Brasil Ltda. (1998 a 2007), subsidiária da Intex Consulting GmbH de Wuppertal, Alemanha, produtora ERP para indústrias têxteis. E-mail: alberto.medeiros@mackenzie.br Gilberto Perez Universidade Presbiteriana Mackenzie, Brasil, Doutor (2007) e Mestre (2003) em Administração pela Universidade de São Paulo, é graduado em Engenharia Eletrônica e Telecomunicações pela Universidade São Judas Tadeu (1982/83) e em Física pela Universidade de São Paulo (2001). Possui Pós-Graduação em Capacitação Gerencial pela FEA/FIA - USP (1999); MBA em Conhecimento, Tecnologia e Inovação pela FEA/FIA - USP (2001). Professor do Programa de Pós-Graduação em Administração (PPGA) na disciplina Inteligência Competitiva do Centro de Ciências Sociais e Aplicadas da Universidade Presbiteriana Mackenzie. Pesquisador do NEPEI - Núcleo de Estudos e Pesquisa em Estratégia, Empreendedorismo, Inovação e Informação da Universidade Presbiteriana Mackenzie. Email: gperez@usp.br Sérgio Lex Universidade Presbiteriana Mackenzie, Brasil, Doutor (2008) e Mestre (1999) em Administração de Empresas pela Universidade Presbiteriana Mackenzie, é graduado em Engenharia Mecânica pela Universidade Presbiteriana Mackenzie (1970). Professor titular e Decano de Extensão e Educação Continuada da Universidade Presbiteriana Mackenzie. Pesquisador do NEPEI - Núcleo de Estudos e Pesquisa em Estratégia, Empreendedorismo, Inovação e Informação o da Universidade Presbiteriana Mackenzie. E-mail: sergio.lex@mackenzie.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

278 Medeiros Jr., A. de, Perez, G., Lex, S.

software, com especial ênfase aos Sistemas Integrados de Gestão. Para a realização da pesquisa, adotaram-se procedimentos de natureza qualitativa em que os 18 fatores, considerados relevantes para uma boa seleção de ERP, foram classificados com a Técnica Delphi, utilizados como entrada em um SAD: Processo de Rede Analítica (Analytic Network Process - ANP) e aplicados como Estudo de Caso em uma empresa de pequeno porte que contratou ERP. Os resultados obtidos demonstraram que o ANP mostrouse eficiente em inter-relacionar critérios e avaliar o alinhamento estratégico entre o Negócio e a Tecnologia de Informação. Palavras-chave: Seleção de sistema de informação, Enterprise Resource Planning ERP, Analytic Network Process – ANP.

1. INTRODUÇÃO Os Sistemas Integrados de Gestão, também conhecidos por ERP (sigla em inglês para Enterprise Resource Planning), formam um conjunto de aplicações que permite às empresas automatizar e integrar substancial parcela de seus processos, envolvendo finanças, controles, logística, suprimentos, manufatura, vendas e recursos humanos, possibilitando o compartilhamento de dados e a uniformização de processos de negócios, além de produzir e utilizar informações em tempo real (Turban, Leidner, Mclean, & Wetherbe, 2010; Laudon & Laudon, 2011; O’Brien & Marakas, 2013). Por apresentarem elevado custo de licenciamento e de implantação e alto grau de incerteza quanto aos resultados, a aquisição de um Sistema Integrado de Gestão gera insegurança nos agentes decisores no momento de selecionar um sistema para as organizações sendo, por isso, uma decisão complexa e não estruturada (Turban, et al., 2010). São frequentes e conhecidos pela comunidade de Tecnologia da Informação (TI) os casos de insucesso na implantação de sistemas de gestão em grandes corporações. Segundo Ganly (2011), estudos feitos pelo Gartner Research demonstram que de 20% a 35% das implementações falham e que o resultado de até 80% dos projetos é questionável por exceder o tempo e/ou orçamento propostos. Soluções mais bem desenvolvidas e mais amigáveis na parametrização, melhor treinamento dos consultores, maior consciência das empresas quanto aos Fatores Críticos de Sucesso – FCS- na implantação dos sistemas têm colaborado para redução de falhas. Pela elevada quantidade de projetos questionáveis, no entanto, deduz-se que existem ainda inúmeras lacunas a serem preenchidas, evidenciando a necessidade de melhor gestão tanto do planejamento de implantação quanto da seleção no momento da aquisição do sistema. Ganly (2011) considera as seis principais armadilhas na implantação de um ERP: 1) escopo inadequado ou deficiente de ERP; 2) ausência do comprometimento dos gestores; 3) orçamento insuficiente ou inadequado; 4) administração e treinamento inadequados; 5) gestão e time de projeto inexperientes; 6) modificações extensivas. Com exceção do segundo item, os demais decorrem, mesmo que indiretamente, de projetos mal elaborados e de aquisição mal encaminhada.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 279 de Negócio

Com este artigo buscou-se, então, responder o seguinte problema de pesquisa: “Como formular um procedimento de decisão para a seleção de um Sistema Integrado de Gestão alinhado ao negócio que possibilite inter-relacionar critérios de avaliação?”. A relevância do tema torna-se evidente uma vez que há ainda uma quantidade considerável de empresas que não utilizaram pacotes de software ERP para integrar os dados e processos de seus departamentos em um sistema único. Estudos do Centro de Estudo Sobre as Tecnologias de Informação e Comunicação (CETIC, 2011), órgão do Comitê Gestor da Internet, demonstram que a proporção dessas empresas representou, em média, 61% do total das empresas em 2010 e, mesmo nas empresas com mais de 250 funcionários, 27% não o utilizam (tabela 1). Tabela 1 - Proporção de empresas que utilizaram pacotes de software ERP para integrar os dados e processos de seus departamentos em um sistema único. Porte

Sim

Não

NS/NR

Total

10 a 49

50 a 249

250 ou mais

Fonte: Adaptado de CETIC (2011).

Este artigo tem como objetivo geral propor um modelo suportado por Sistema de Apoio à Decisão (SAD), que possibilite às empresas analisar, quando surge o interesse de aquirir um Sistema Integrado de Gestão (ERP), qual das ofertas disponíveis estará mais adequada às suas estratégias de negócio. 2. REFERENCIAL TEÓRICO 2.1 Investimentos em Sistemas e Tecnologia da Informação Segundo Perez (2007), as organizações da era da informação e do conhecimento têm investido cifras cada vez maiores em Sistemas de Informação (SI) e Tecnologia da Informação (TI), uma vez que, gradativamente, os SIs e as TIs vêm se tornando um componente significativo em praticamente tudo o que as empresas fazem (Turban et al., 2010). Autores como Lunardi, Becker e Maçada (2003) e Perez (2007) relatam que alguns setores têm investido significativas quantias em SI e TI e apontam a concorrência e a rivalidade como sendo os principais fatores que justificam essa prática. Outro fator relacionado com o investimento em SI e TI é a busca por competitividade e retornos acima da média. Os investimentos em SI e TI para a obtenção de melhores condições de segurança têm aumentado substancialmente nos últimos anos, especialmente, no setor financeiro. Applegate, Austin e McFarlan (2003) consideram que os investimentos em SI e TI podem contribuir para a gestão do conhecimento e os resultados obtidos nesse caso JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

280 Medeiros Jr., A. de, Perez, G., Lex, S.

são: a melhoria do desempenho do conhecimento dos colaboradores da empresa e o aprendizado organizacional. 2.2 Os Sistemas de Gestão Integrados (ERP) Os Sistemas de Informação exercem três importantes papéis em todos os tipos de organização (O’Brien & Marakas, 2013): 1) suportar as operações; 2) suportar as estratégias competitivas; 3) suportar as decisões. O suporte às operações é obtido principalmente por sistemas transacionais. O suporte às estratégias competitivas pode-se dar pelos usos inovadores de sistemas de informação. O suporte à decisão, finalmente, é feito por sistemas de informação gerencial ou executiva, por recursos de armazenamento e mineração de dados (Data Warehouse/Data Mining) ou pelos SAD (O’Brien & Marakas, 2013). Dentre os sistemas transacionais destaca-se o Sistema Integrado de Gestão que pode ser conceituado como sendo um pacote de software, abrangente e integrado, que possibilita a padronização e a automação de processos de negócio utilizando uma base de dados unificada e transações em tempo real. A sua implementação nas empresas acaba com as “ilhas de automação” (McFarlan & McKenney, 1982) existentes antes disso. 2.2.1 Motivos para Adoção de ERP A opção de se adotar um ERP, apesar de ser um desejo usual dos altos executivos de todas as empresas, sempre é uma decisão complexa. No entanto, observa-se ser frequente uma expectativa acima daquilo que as aplicações geralmente oferecem. Colângelo (2001) considera que há três classes de motivos que levam uma empresa a implantar um ERP:  Negócios: visa elevar a lucratividade ou o fortalecimento da posição competitiva da organização;  Legislação: visa atender às exigências legais as quais a empresa deve cumprir e que não são atendidas pelos sistemas legados (aqueles em uso na empresa);  Tecnologia: visa atender às mudanças necessárias decorrentes da obsolescência econômica das tecnologias em uso ou às exigências de parceiros de negócios. Há, no entanto, algumas expectativas mínimas comuns (Koch, 1999; Colângelo, 2001), destacando-se dentre outras: a) informações gerenciais ou executivas; b) informações financeiras integradas; c) redução de inventário e d) falta de integração entre os sistemas legados. 2.2.2 Fatores Críticos de Sucesso na Implantação de ERP Os Fatores Críticos de Sucesso (FCS) representam um mecanismo de identificação das necessidades de informação pelos gestores das organizações. Para Rockart (1979, p. 85), “os FCS são, para quaisquer negócios, o número limitado de áreas nas quais os resultados, se satisfatórios, asseguram o desempenho competitivo bemsucedido para a organização”. São as poucas áreas-chave onde “as coisas têm que dar certo” para que o negócio dê certo. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 281 de Negócio

Esteves-Sousa e Pastor-Collado (2000) reuniram em pesquisa os FCS encontrados na literatura em dez pesquisas a partir de análise de relatórios de implantação de Sistemas Integrados de Gestão. Determinaram as similaridades e os padrões comuns entre eles e propuseram a sua unificação conforme o modelo apresentado na figura 1.

Tecnológico

Organizacional

Nível Perspectiva

FCS

Apoio contínuo da administração. Administração de efetivas mudanças organizacionais. Administração de um bom escopo de projeto. Composição adequada da equipe de projeto. Estratégico Processo abrangente de reengenharia de negócios. O papel do patrocinador do projeto. Participação e envolvimento dos usuários. Confiabilidade entre os parceiros. Dedicação do grupo de trabalho e consultores. Forte comunicação interna e externa. Projeto e programação de implantação formal. Programa adequado de treinamento. Tático Localização preventiva de problemas. Uso apropriado dos consultores. Decisores autorizados. Estratégia adequada de implantação. Evitar customizações. Estratégico Versão adequada do sistema. Configuração adequada do software. Tático Conhecimento dos sistemas legados. Figura 1. Relevância dos FCS por perspectiva. Fonte: Esteves-Sousa e Pastor-Collado (2000).

2.2.3 Seleção de ERP A seleção de fornecedores é uma tarefa complexa e exige esforço por parte do contratante sob pena de fracasso e geração de ambiente de animosidade na relação cliente-fornecedor, caso a seleção não obedeça a critérios pré-estabelecidos (Perez & Zwicker, 2005). Desta forma, mais do que buscar fornecedores, as empresas devem buscar relações duradouras de parceria com seus fornecedores. Perez e Zwicker (2005) enfatizam que a capacitação tecnológica, a perspectiva de um relacionamento duradouro, a postura ética, a excelência dos serviços prestados, a capacidade de expansão de futuras ofertas de soluções, são, dentre outros, fatores relevantes e que devem ser levados em conta ao se escolher um fornecedor. Por suas características, a seleção de um aplicativo como o ERP deve seguir critérios previamente definidos pelos gestores tomadores de decisão. Mendes e Escrivão Filho (2007) apresentam um roteiro considerado ideal para atender a aquisição de Sistemas Integrados de Gestão e possibilita a avaliação das adequações e impactos nas mudanças da organização. O estudo apresentado, no entanto, não orienta sobre a escolha do sistema ou fornecedor. O processo de seleção usualmente inicia-se pela emissão de uma requisição de proposta conhecida em inglês por de Request for Proposal – RFP-, que é um convite aos potenciais fornecedores para que eles submetam à organização as ofertas para produtos ou serviços específicos. Esse documento é um questionário que envolve os requiJISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

282 Medeiros Jr., A. de, Perez, G., Lex, S.

sitos da organização, não apenas quanto ao preço, mas deve incluir ainda informações mais profundas sobre a fornecedora de produto ou serviço, tais como: dados financeiros, competência técnica, previsão de implantação, referências comerciais e principalmente um detalhamento sobre o produto a fim de se verificar o seu grau de aderência aos negócios da empresa compradora. Os requisitos variam de empresa a empresa e, não raro, chegam a centenas de itens a serem analisados. 2.2.3.1 Critérios de Seleção de ERP

1. Critérios técnicos 2. Funcionalidade 3. Referências do fornecedor 4. Custo 5. “Implantabilidade” 6. Facilidade de customização 7. Integração modular cruzada 8. Consultorias de seleção e implantação 9. Ajuste estratégico 10. Riscos 11. Flexibilidade 12. Análise de benefícios 13. Serviço e suporte 14. Confiabilidade do sistema 15. Melhor ajuste com a estrutura organizacional 16. Ajuste com sistema de matriz e/ou parceiro 17. Método de implantação do software 18. Domínio de conhecimento do fornecedor 19. Posição do fornecedor no mercado 20. Compatibilidade com outros sistemas 21. Visão do fornecedor 22. Mudanças administrativas 23. Tempo de implantação 24. “Escalabilidade” para permitir crescimento 25. Atualização da tecnologia 26. Facilidade de uso 27. Segurança 28. Localização

X X X X X

X X X X X X X X

X X X

Corrêa (1995)

Fitzgerald (1998)

Illa et al (2000)

X X X X

X X

X X X X X

X X X X

X X X X X X X X X X X X X

X X X X X X X X

X X

Figura 2: Critérios de Seleção Identificados Fonte: Baseado na Literatura Pesquisada JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

Kumar et al (2002)

Baki et al (2005)

Teltumbde (2000)

A figura 2 apresenta o resultado de pesquisa feita na literatura existente sobre os critérios mais frequentemente considerados nas avaliações. Esses critérios podem ser utilizados em SAD, com a finalidade de se avaliar qual ERP seria mais adequado à empresa adquirente.

www.jistem.fea.usp.br

X X X X X

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 283 de Negócio

2.4 O alinhamento da Estratégica de TI com a Estratégia Corporativa Segundo Chan e Huff (1993), o alinhamento estratégico é a integração da TI com as estratégias fundamentais e as competências essenciais de uma organização. Henderson e Venkatraman (1993) desenvolveram um modelo para analisar alinhamento da estratégica de TI com a estratégia corporativa, o qual apresenta quatro blocos que representam os quatro domínios da escolha estratégica (figura 3).

Figura 3. Modelo de alinhamento estratégico Fonte: Adaptado de Henderson e Venkatraman (1993).

O par de blocos à esquerda foca o Negócio, enquanto o par da direita foca a Tecnologia de Informação. Por outro ponto de vista, o par superior (dos domínios externos) representa a Estratégia, enquanto o par inferior (dos domínios internos) representa a Infraestrutura. O inter-relacionamento é tal que qualquer decisão estratégica envolve decisões em um ou mais domínios. Um domínio orienta ou impacta atividades em um ou mais dos outros. A estratégia de negócio, por exemplo, impacta a infraestrutura organizacional e a estratégia de TI impacta a infraestrutura de TI. Este modelo permite às organizações analisarem em que domínios elas são fortes ou fracas e, quais os efeitos de uma decisão em um domínio sobre os demais. Apesar de o alinhamento estratégico ser feito geralmente no domínio externo, ou seja, na relação entre as estratégias de negócio e de TI, isto não significa que o domínio interno (relação entre as infraestruturas organizacional e de TI) seja secundário ou menos importante.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

284 Medeiros Jr., A. de, Perez, G., Lex, S.

Henderson e Venkatraman (1993) consideram dois tipos de integração entre os domínios de negócio e de TI. Um é a integração estratégica, que vincula a estratégia de negócio à estratégia de TI, refletindo os domínios externos. Outro é a integração funcional, refletindo os domínios internos, que vincula a infraestrutura organizacional à infraestrutura de TI. Cada domínio contém ainda três conjuntos de escolha, conforme identificados na figura 3 e descritos na figura 4. Henderson e Venkatraman (1993) consideram que o alinhamento estratégico é um resultado simultâneo tanto de um ajuste estratégico quanto de uma integração funcional. O ajuste estratégico é obtido quando se tomam decisões que posicionam a empresa adequadamente no mercado, quando no domínio do negócio, ou as tecnologias na empresa, quando no domínio de TI. Um bom ajuste estratégico permite capitalizar a estrutura, processos e habilidades das pessoas na implementação da estratégia organizacional, no domínio dos negócios ou de tecnologia, no domínio de TI (Luftman, Lewis, & Oldach 1993; Curtin, 1996). Negócio

Tecnologia de Informação

Escopo do negócio:

Escopo de tecnologia:

São as decisões e escolhas que definem o campo de concorrência da empresa como: produtos, nicho, clientes, etc.. Inclui as forças competitivas de Porter (1989). São perguntas típicas: qual é o nosso negócio? quais são os nossos produtos, serviços e mercados-alvos?

Estratégia

Competências distintivas: São as áreas que determinam como as empresas vão competir na entrega de seus produtos ou serviços. É a aptidão da empresa em diferenciar seus produtos ou serviços da concorrência. A pergunta típica é: no que devemos nos concentrar em fazer melhor a fim de nos distinguirmos de nossos concorrentes? Governança do negócio: São as decisões referentes a eventuais parcerias ou a terceirizações visando obter vantagens de escala para o atendimento de um mercado em particular. A pergunta típica é: de que relações com negócios externos e/ou joint ventures nós dependemos?

Especifica as tecnologias críticas responsáveis pelo sucesso da organização, tais como sistemas de base de conhecimento, robotização, multimeios, etc. A pergunta típica é: que tecnologias tanto suportam quanto criam oportunidades de negócios estratégicos? Competências sistêmicas: Características importantes de TI críticas na criação ou extensão das estratégias de negócios, tais como conectividade, acessibilidade, confiabilidade etc.. A pergunta típica é: que características de TI criam vantagens de negócios?

Governança de TI: Definição da propriedade da tecnologia (usuário final, comitê gestor, etc.) ou da possibilidade de alianças tecnológicas como parcerias, terceirizações ou ambas, ou ainda decisões do tipo “comprar ou fazer” (make-or-buy). A pergunta típica é: de quais relacionamentos externos como terceirização, decisões comprar ou fazer etc., nós dependemos?

Figura 4. Modelo de alinhamento estratégico

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 285 de Negócio

Negócio Infraestrutura administrativa:

Infraestrutura

São escolhas que estabelecem a estrutura de gestão e de processo de trabalho que uma empresa vai operar. É a estrutura de regras, responsabilidades e poder da empresa. As perguntas típicas são: qual é a nossa estrutura organizacional? quem se reporta a quem? Processos: São escolhas que determinam como as funções básicas do negócio irão operar ou fluir. Definem o quanto os fluxos de trabalho serão reestruturados ou integrados para melhorar a eficiência e a eficácia das empresas. A pergunta típica é: quais são os nossos processos-chave de negócio? Habilidades: São escolhas onde se devem definir claramente os recursos humanos envolvidos na estratégia e a eventual necessidade de contratação de serviços terceirizados. A pergunta típica é: que recursos humanos nós temos (ou necessitamos) para acompanhar nossas concorrências específicas?

Tecnologia de Informação Arquiteturas: São as escolhas, prioridades ou políticas que sintetizam as aplicações, dados, software e hardware em uma plataforma coesa. A pergunta típica é: quais são as nossas opções de plataformas, hardware, software, configurações de rede e arquitetura de dados? Processos: É a configuração quanto às práticas de desenvolvimento de aplicações e controles gerenciais dos sistemas. A pergunta típica é: quais são os processos de TI: desenvolvimento, manutenção, operações de sistema, administração de base de dados? Habilidades: São as experiências, competências, comprometimentos, valores e normas que regem as tarefas no objetivo da produção e entrega de produtos e/ou serviços de TI. A pergunta típica é: quais são as habilidades que os nossos gestores de TI e pessoal de apoio necessitam para manter a arquitetura e executar os processos?

Figura 4. Modelo de alinhamento estratégico (continuação) Fonte: Adaptado de Luftman, Lewis e Oldach (1993) e Curtin (1996).

2.5 Sistemas de Apoio à Decisão Segundo Shimizu (2010), existem vários métodos que possibilitam o apoio à decisão, dos quais se destacam os seguintes:  Teoria da utilidade: assume-se que um decisor sempre busca a solução que gere a maior satisfação ou “utilidade” para ele.  Métodos ELECTRE (sigla em francês para Elimination et Choix Traduisant La Realité): trata-se de um algoritmo de decisão para problemas com múltiplos critérios, que reduz o tamanho do conjunto de alternativas possíveis, classificando-as conforme o critério de dominância de uma sobre a outra.  Método MACBETH (sigla em inglês para Measuring Attractiveness by a Categorical Based Evaluation Technique): permite concentrar os diversos critérios de avaliação em um critério único de síntese pela atribuição de pesos aos vários critérios, respeitando as opiniões dos decisores.  Método do processo de hierarquia analítica (Analytic Hierarchy Process - AHP): o seu pressuposto básico é que um problema complexo pode ser eficientemente resolvido quando é decomposto em diversas partes interligadas por uma estrutura hierárquica, determinando-se

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

286 Medeiros Jr., A. de, Perez, G., Lex, S.

pesos específicos para cada um dos critérios - comparados par a par - para que ocorra o cotejo entre alternativas.  Método do processo de rede analítica (Analytic Network Process - ANP): possibilita ao decisor considerar a possível existência de dependências entre os fatores de decisão e analisar o efeito e retroalimentação decorrentes dessas dependências. Isso é obtido por julgamentos e uso de medições por escalas proporcionais. Enquanto o AHP é a teoria que depende dos valores e dos julgamentos dos indivíduos e grupos, baseado em um dado cenário, o ANP é a generalização do AHP (Saaty, 2001), pois permite a construção de diversos cenários, para problemas complexos de decisão. A principal diferença entre o AHP e o ANP é que este tem uma abordagem que substitui as hierarquias por redes sendo que, em ambas as abordagens de tomada de decisão, os julgamentos são executados conjuntamente e de uma forma organizada para produzir prioridades. No ANP, a interação e a retroalimentação dão-se em agrupamentos (clusters) de elementos com dependência interna e entre esses agrupamentos com dependência externa, capturando melhor os “efeitos complexos das reciprocidades nas sociedades humanas, especialmente quando ocorrem riscos e incertezas” (Saaty, 2001). O ANP, por possibilitar o feedback do julgamento e por sua característica de possibilitar agrupamentos, foi escolhido como instrumento de pesquisa dos Estudos de Caso, uma vez ser o ANP um método multicritérios relativamente simples e com uma abordagem intuitiva que pode ser aceita facilmente por administradores e outros decisores (Presley & Meade, 1999). A sua aplicação que pode ser feita com a utilização de planilhas eletrônicas, como o Excel, mas permite também a facilidade de uso pela aplicação do software SuperDecisions (www.superdecisions.com). O feedback possibilita julgar não apenas quanto à importância de cada alternativa para cada critério como estes têm a sua importância julgada para cada alternativa. Não se pretendeu, neste estudo, comparar SADs. O objetivo foi verificar se há a possibilidade de se obter uma seleção de ERP alinhada às estratégias de negócio utilizando um SAD. Por isso, foi definida, ainda que de forma prescritiva, a utilização do ANP, uma vez que ele possibilitou que a formação do grid, conforme Henderson e Venkatraman (1993), pudesse ser reproduzido na forma dos seus agrupamentos. Esses agrupamentos contêm os critérios a serem julgados conforme as dependências entre os fatores de decisão e analisar o efeito e retroalimentação decorrentes dessas dependências. Isso é obtido por julgamentos e uso de medições por escalas proporcionais para a distribuição de influência entre os fatores e os grupos de fatores na decisão, possibilitando a alocação de recursos conforme as escalas proporcionais de prioridades. Os julgamentos no ANP são feitos par a par utilizando uma escala fundamental (também utilizada no AHP), onde a intensidade 1 indica a mesma importância entre as duas atividades, 3 indica importância moderada, 5 indica forte importância, 7 indica muito forte importância e 9 indica extrema importância. As intensidades pares indicam importâncias intermediárias entre as ímpares e os valores recíprocos são utilizados se JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 287 de Negócio

uma primeira atividade comparada a uma segunda atividade tem valor igual a um dos valores de intensidade indicados. e Então, esta segunda atividade tem o valor de intensidade recíproco à primeira (Saaty, 2000, 2001, 2005). Conforme Saaty (2005, p.47), duas perguntas devem ser respondidas: “1) Dado um critério do qual entre dois elementos é o mais dominante quanto ao critério?; 2) Qual dos dois elementos influencia mais um terceiro elemento quanto ao critério?” O mais simples modelo tem um agrupamento denominado “objetivo” (goal), contendo um elemento “objetivo”, um agrupamento denominado “critério” (criteria) contém os elementos “critérios” e um agrupamento denominado “alternativa” (alternative) contém os elementos de “alternativas”, conforme a figura 5. Quando os agrupamentos são conectados por uma linha significa que os nós (elementos) estão conectados (Saaty, 2001). A direção da seta indica a relação de comparação dos agrupamentos. Desta maneira, uma seta com duplo sentido indica que há influência dos subcritérios nas alternativas e vice-versa. A seta em forma de arco sob o grupo “alternativas” (indicando um loop) significa que os elementos nele contidos se influenciam. Os elementos contidos em “critérios” e em “subcritérios” foram considerados independentes. O desempenho de uma alternativa pode influenciar o de outra alternativa. O desempenho de um critério em um subcritério, no entanto, não depende do desempenho em outro subcritério (Salomon, 2004).

Figura 5, Representação abstrata dos componentes de uma decisão em forma hierárquica. Fonte: Adaptado de Saaty (2005).

2.5.1 Componentes Fundamentais do ANP 2.5.1.1 Dependências Em um ANP os componentes podem influenciar outros elementos no mesmo componente. Isso é chamado dependência interna (inner dependence) ou com outros componentes, chamado dependência externa (outer dependence). Na figura 6 a linha entre os componentes C4 e C2 indica uma dependência externa quanto às propriedades comuns dos elementos em C2 quanto aos elementos em C4, enquanto que os loops nos componentes C1 e C3 indicam dependência interna quanto às propriedades comuns dos elementos nesses componentes (Saaty, 2005).

Figura 6. Representação abstrata das dependências em uma rede de decisão.

Fonte: Saaty (2005). Para evitarem-se redes muito complexas, o ANP admite que um agrupamento seja composto de uma sub-rede, a qual contém novos agrupamentos e respectivos nós.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

288 Medeiros Jr., A. de, Perez, G., Lex, S.

2.5.1.2 A Supermatriz A supermatriz consiste em blocos de vetores de prioridades para os agrupamentos na rede, os quais estão dispostos na supermatriz de cima para baixo a partir do seu lado esquerdo. Sob cada agrupamento estão os nós a ele pertencentes, ocorrendo o mesmo com as linhas (Saaty, 2005). A comparação entre todos os agrupamentos vinculados resulta em uma matriz de alcance global, a qual é utilizada para ponderar os blocos de agrupamentos na supermatriz. Enquanto a supermatriz não é ponderada pela matriz de alcance global dos agrupamentos ela é chamada de “supermatriz não ponderada”. A soma dos valores nas colunas será superior a um naquelas que contém algum nó que é comparado a outro em uma dependência interna. Após a ponderação ser aplicada pela matriz de alcance global dos agrupamentos, ela passa a ser uma “supermatriz ponderada”. Neste caso, o efeito da ponderação faz com que a soma de cada coluna da supermatriz tenha o valor um (Saaty, 2005). Se o projeto de ANP for criado com sub-redes, cada uma delas terá a sua própria supermatriz. 3. PROCEDIMENTOS METODOLÓGICOS 3.1 Técnicas de Pesquisa A pesquisa envolveu a combinação de duas técnicas de natureza qualitativa por meio da Técnica Delphi e da estratégia de Estudo de Caso. Com a técnica Delphi procurou-se classificar por consenso de um grupo de especialistas os critérios que deveriam ser considerados mais apropriados na seleção de um Sistema Integrado de Gestão, enquanto a segunda visou verificar a utilização do procedimento em uma empresa de pequeno porte. 3.1.1 Técnica Delphi (TD) É uma abordagem de pesquisa exploratória (Hair, Babin, Money, & Samouel, 2005). Trata-se de uma ferramenta de análise qualitativa usada principalmente em situações de carência de dados históricos ou de rupturas tecnológicas. Um importante aspecto da TD é ser uma abordagem que procura obter o consenso de um grupo de especialistas (Turoff & Linstone, 1975; Wright, 1986). Na primeira parte da pesquisa foi utilizada a TD revisada, conforme proposta por Dickson e Nechis (1984), priorizando os pontos-chave em SI para classificar os critérios relevantes para a avaliação do ERP. 3.1.2 Estudo de Caso Dentre as diversas estratégias de pesquisa qualitativa observa-se que o Estudo de Caso é uma das técnicas das quais têm sido utilizadas com certa frequência para as pesquisas organizacionais por apresentar uma possibilidade de reflexão dos resultados na área das ciências administrativas. Trata-se de uma técnica que envolve o estudo intensivo e detalhado de uma entidade bem definida: o “caso” (Coutinho & Chaves, 2002). Segundo Yin (2010, p. 19), o Estudo de Caso “é uma estratégia preferida [...] JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 289 de Negócio

quando o pesquisador tem pouco controle sobre os acontecimentos e quando o foco se encontra em fenômenos contemporâneos inseridos em algum contexto da vida real”. 4. APRESENTAÇÃO E ANÁLISE DOS RESULTADOS 4.1 Modelo do Instrumento de Pesquisa Utilizado Para que houvesse uma redução de fatores evitando-se utilizar todos os 28 critérios apresentados na figura 2, o que representaria uma quantidade de comparações par a par que inviabilizaria a aplicação do modelo, os mais relevantes foram obtidos a partir de duas rodadas de Técnica Delphi aplicadas inicialmente em 41 especialistas no assunto, dos quais 27 apresentaram respostas válidas. Com isso, definiu-se que no ANP seriam utilizados os 18 critérios de avaliação mais relevantes, apontados pelos especialistas (figura 7), os quais foram então distribuídos entre os agrupamentos criados no software SuperDecisions conforme as escolhas propostas por Henderson e Venkatraman (1993). Sub-rede

Escolha

Escopo de TI

Estratégia de TI

Critério Flexibilidade

Denota a habilidade do sistema em suportar as necessidades do negócio durante o seu tempo de vida.

Funcionalidade

Tem três aspectos principais: 1) quais áreas funcionais o produto pode cobrir; 2) o quanto o produto é flexível quanto à adaptabilidade e abertura; e, 3) características específicas de alguns ERP.

Confiabilidade do sistema

O sistema deve incorporar as melhores práticas de negócio de cada área, bem como as mais recentes tendências de TI. Essa confiabilidade deve ser verificada com os usuários.

Custo

O preço de um ERP geralmente é muito alto. O custo total de propriedade (TCO – Total Cost Ownership) do sistema deve incluir o preço das licenças, a sua manutenção e atualizações, softwares complementares, o hardware, rede, consultoria, treinamento, equipe de implantação e ainda outros custos.

Facilidade de customização

Ainda que se procure evitar a customização, a maioria das empresas necessita dela, uma vez que há a necessidade em se adequar uma solução genérica às necessidades específicas da empresa. A customização pode ser feita internamente ou por consultorias especializadas ou ser desenvolvida pela própria produtora do sistema, que passa a ser incorporada à aplicação principal.

“Implantabilidade”

Alguns projetos são mais difíceis de serem implantados que outros. Alguns talvez nunca consigam efetivamente ir adiante. As implantações de sistemas ERP exigem também, com frequência, redesenho dos processos da empresa, tanto para que não se automatizem processos imperfeitos como para que os processos passem a se adequar ao melhor uso do sistema.

Referências do fornecedor

As referências de vendas, a reputação e a internacionalidade do vendedor, com histórico de projetos bem-sucedidos, devem ser considerados importantes critérios para o processo de seleção.

Segurança

Os aspectos relacionados à segurança e à integridade dos dados devem ser considerados quando da aquisição de um sistema. Destacam-se os recursos como alocação e gestão de senhas de acesso, proteção contra invasores externos, apoio a rotinas de cópia de segurança e recuperação de dados, recursos para lidar com quedas de energia durante execução de rotinas.

Tempo de implantação

A implantação de um ERP é altamente custosa e complexa em projetos de grande porte. Ele está fortemente correlacionado à estratégia de implantação, podendo também ocorrer por mudanças no escopo. Soluções específicas podem também reduzir o tempo de implantação.

Ajuste com sistema de matriz e/ou parceiros

A compatibilidade com a matriz ou parceiros pode afetar no processo de decisão de algumas empresas. Esta compatibilidade pode também afetar o sucesso do projeto.

Compatibilidade

Nenhuma aplicação pode fazer tudo que uma empresa necessita. Não existe aderência absoluta. A solução selecionada deve ter recursos para integração

Competências sistêmicas

Governança de TI

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

290 Medeiros Jr., A. de, Perez, G., Lex, S.

Sub-rede

Escolha

Critério com outros sistemas

com os sistemas legados da empresa e com outros softwares especializados que ela utiliza para cobrir todas as suas necessidades.

Critérios técnicos

O ERP é uma aplicação de TI e as suas dimensões tecnológicas assumem importância em ambientes de rápidas mudanças tecnológicas e, por isso, deve estar atualizado nas tendências de TI, particularmente quanto aos sistemas gerenciadores de banco de dados, ambiente cliente-servidor, hardware, sistema operacional, etc..

Escalabilidade para permitir crescimento

A escalabilidade permite ao ERP crescer progressivamente, na medida do gradual crescimento dos usuários na empresa, permitindo na mesma proporção o aumento da capacidade das plataformas de hardware e software, evitando descontinuidades bruscas de trocas completas de plataforma.

Configuração adequada do software

Envolve a adaptação das funcionalidades genéricas de um pacote às necessidades particulares de uma organização.

Integração modular cruzada

O real benefício de um sistema ERP é a integração. E esta integração deve ser completa entre os módulos. Se ela não existir, além de elevar o custo de implantação, pode afetar a eficiência do sistema que pode decrescer.

Consultorias de seleção e implantação

Os consultores devem ter experiência nas especificidades da empresa, com conhecimento abrangente dos módulos e ser capaz de determinar quais são os recursos do sistema que melhor atenderão a empresa.

Domínio de conhecimento do fornecedor

É importante que o desenvolvedor do sistema tenha conhecimento do ramo de negócio. Se for uma manufatura, a empresa precisa encontrar um software de um fornecedor especializado em sua área de atuação.

Serviço e suporte

Uma instalação completa de ERP pode alcançar um investimento muitas vezes superior ao custo de um software. Por isto, o serviço e o suporte associados tornam-se fatores vitais para o sucesso do negócio entre o usuário e o fornecedor de um sistema.

Arquiteturas de TI

Infraestrutura de TI

Processos de TI

Habilidades de TI

Figura 7. Critérios para avaliação de ERP atribuídos às escolhas no modelo de alinhamento estratégico nas sub-redes subordinadas à Tecnologia de Informação (Continuação).

A figura 8 apresenta o resultado, considerando-se haver dependência entre e intraagrupamentos, representados pelas setas e pelos arcos.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 291 de Negócio

a. Rede inicial Sub-redes

b. Estratégia de Negócio sob o agrupamento Negócio

d. Infraestrutura de Negócio sob o agrupamento Negócio

c. Estratégia de TI sob o agrupamento TI

e. Infraestrutura de TI sob o agrupamento TI

Figura 8. Representação gráfica da rede e sub-redes na escolha de ERP alinhado à estratégia corporativa utilizadas como instrumento de pesquisa no Estudo de Caso.

O modelo foi reproduzido no software SuperDecisions, cujos recursos gráficos facilitam a compreensão e o controle do decisor. Foi elaborado um protocolo de pesquisa com todas as possibilidades de julgamentos par a par e aplicado em um Estudo de Caso, com o gestor de uma empresa que havia participado da decisão da escolha do seu ERP, dentre três opções (ERP #1, ERP #2, ERP #3) oferecidas pelo mercado. Escolheu-se uma empresa de pequeno porte, produtora de acessórios para a indústria automobilística. O Estudo de Caso visou identificar: 1) se o ANP é adequado como ferramenta para análise para seleção de ERP; 2) se o uso do ANP construído como o modelo de alinhamento estratégico (modelo de Henderson e Venkatraman) possibilita que a avaliação considere as importâncias de Negócio e de TI atribuídas à avaliação. 4.2 Resultados Obtidos Na empresa escolhida foi avaliado com o ANP o impacto do julgamento entre Negócio e TI para verificar se refletiria a necessidade de alinhamento entre eles. Foram efetuadas diversas simulações, ficando, então, confirmado o impacto do alinhamento estratégico entre o Negócio e a Tecnologia de Informação, conforme contemplado no modelo de Henderson e Venkatraman (1991).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

292 Medeiros Jr., A. de, Perez, G., Lex, S.

A figura 9 apresenta em gráfico a simulação do impacto da importância dada pelo decisor da empresa, onde se observa que o ERP melhor classificado após a avaliação no ANP (ERP #3) tem sempre o valor igual a 1, já que este é o resultado ideal. A simulação mostra que, se o decisor considerar que a TI tem maior importância do que o Negócio, então a possível decisão pelo ERP #1 aproxima-se bastante do ERP #3.

Figura 9. Resultado ideal (onde o ERP com melhor resultado ERP #3 tem valor igual a 1) demonstrando que, se o decisor considerar maior intensidade em TI, o ERP #1 se aproxima do ERP #3.

Foram feitas as análises de sensibilidade considerando-se os resultados das variáveis dependentes (os ERPs) para as variáveis independentes (os agrupamentos) e também refletiram isso para as intensidades de julgamento da importância do Negócio relativamente à importância de TI na organização. A análise de sensibilidade trabalha com cenários hipotéticos do tipo “what...if”, onde o valor de apenas uma variável é alterada repetidas vezes, e são observadas as alterações ocorridas nas outras variáveis. Isso é feito por repetidas mudanças em apenas uma variável por vez. Essa análise possibilita a compreensão do impacto dessa variável sobre as demais (O’Brien & Marakas, 2013). Saaty (2000, p. 112) considera que “uma preocupação útil em qualquer teoria baseada em medições é efetuar hipoteticamente tanto uma pequena quanto uma grande perturbação nos valores das medidas e observar os seus efeitos na saída”. A figura 10 apresenta essas sensibilidades de forma gráfica, extraídas do software SuperDecisions. Observe-se que, apesar de o ERP #3 apresentar-se com tendência crescente em relação direta aos experimentos em Estratégia de Negócio, isso não ocorre em Estratégia de TI, no qual, a partir do grau 0,5 dos experimentos, o ERP #1 passa a se classificar na primeira posição. Na Infraestrutura de TI o ERP #1 tem vantagem sobre o ERP #3 quando a TI é avaliada como de extrema importância ao ser comparada ao Negócio, porém essa posição inverte-se a partir do grau 0,3 dos experimentos. Isso demonstra que o modelo é sensível, pois, segundo Turban et al. (2010), um modelo é sensível quando pequenas mudanças nas condições determinam soluções diferentes.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 293 de Negócio

1. Variável independente: Estratégia de Negócio

2. Variável independente: Estratégia de TI

3. Variável independente: Infraestrutura de Negócio

4. Variável independente: Infraestrutura de TI

a. Extrema importância de Negócio relativa à TI,

b. Negócio e TI têm a mesma importância.

Figura 10. Análise de sensibilidade.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

c. Extrema importância de TI relativa ao Negócio.

294 Medeiros Jr., A. de, Perez, G., Lex, S.

5. CONCLUSÃO O alto custo e os insucessos de implantação ou de implantações questionáveis têm feito com que a seleção de um Sistema Integrado de Gestão seja um desafio aos executivos das organizações. Uma dificuldade recorrente dos gestores é como interrelacionar os diversos critérios de seleção a serem considerados. Como decorrência, desenvolveu-se este estudo com o intuito de responder ao seguinte problema de pesquisa: Como formular um procedimento de decisão para a seleção de um Sistema Integrado de Gestão alinhado ao negócio que possibilite inter-relacionar critérios de avaliação? Para a sua consecução, os vários critérios selecionados a partir da literatura existente foram posteriormente submetidos a especialistas que, pela utilização da Técnica Delphi, os classificaram conforme o seu grau de relevância. Utilizar os critérios identificados pela pesquisa pode ser um bom início, porém cada empresa tem as suas peculiaridades, as quais devem ser pesquisadas junto aos usuários mestres. Todavia, este procedimento não resulta em uma condição sine qua non. A utilidade e aplicabilidade dos resultados obtidos na pesquisa residem justamente no fato de que dela participaram especialistas no tema Sistemas Integrados de Gestão/ERP, que puderam, de forma transparente classificar os critérios identificados na pesquisa, com base em seus conhecimentos e suas experiências adquiridas ao longo do tempo. Como resultado, as empresas podem utilizá-los como ferramenta de apoio à decisão na compra de sistemas e tecnologias de informação, como é o caso do ERP. A análise da sensibilidade mostra-se bastante útil para se verificar a robustez do modelo, demonstrando que, dependendo das intensidades atribuídas aos agrupamentos tomados como variáveis independentes, pode ocorrer uma efetiva troca de classificação entre os sistemas analisados. Saaty (2000, p. 112) considera que “uma preocupação útil em qualquer teoria baseada em medições é efetuar hipoteticamente tanto uma pequena quanto uma grande perturbação nos valores das medidas e observar os seus efeitos na saída”. Deve-se, no entanto, considerar que nem sempre o sistema adquirido mostra-se alinhado à estratégia corporativa. Alinhar estrategicamente não significa necessariamente considerar igual importância entre o Negócio e a Tecnologia. Um decisor de uma empresa voltada à TI poderá eventualmente considerar Estratégia de TI mais pertinente do que a de Negócio. O Estudo de Caso apresentado neste artigo avaliou o uso do ANP como um SAD que pudesse contemplar essas opções de cada decisor. Os indícios apurados no transcorrer da aplicação dos modelos como instrumento de pesquisa nos Estudos de Caso permitem considerar que o ANP nos casos estudados, assim como no caso apresentado, pode possibilitar os resultados necessários à tomada adequada de decisão. 5.1. Limitações da pesquisa e recomendações A pesquisa não teve como objetivo avaliar SAD, mas sim, avaliar se um desses sistemas, no caso o ANP, poderia contemplar o alinhamento entre o Negócio e a Tecnologia de Informação. Foram definidos, nesta pesquisa, os critérios de escolha apenas para os domínios de Tecnologia de Informação, deixando-se para uma futura pesquisa a definição de critérios para os domínios do Negócio. Os resultados aqui apresentados JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

Utilização da Rede Analítica para a Seleção de Sistemas Integrados de Gestão (ERP) Alinhados à Estratégia 295 de Negócio

são típicos da empresa (caso) estudada. Portanto, não podem ser generalizados. Contudo, eles podem ter valor para empresas e gestores que estão vivenciando a etapa de aquisição/escolha de Sistemas Integrados de Gestão. Recomenda-se, então, que a pesquisa seja continuada adotando-se um número maior de casos ou mesmo pela utilização dos critérios para os domínios do Negócio. Outra possibilidade seria elaborar estudos de natureza quantitativa que busquem mensurar a validade dos critérios de avaliação classificados pelos especialistas. REFERÊNCIAS BIBLIOGRÁFICAS Applegate, L. M., Austin R. D. & McFarlan, F. W. (2003) Corporate Information Strategy and Management – The challenges of managing in a network economy. McGraw-Hill. Baki, B. & Çakar, K. (2005) Determining the ERP package-selecting criteria: the case of Turkish manufacturing companies. Business Process Management Journal. 11, pp. 75-86. Cetic – Centro de Estudos sobre as Tecnologias de Informação e de Comunicação (2011) Pesquisa sobre o uso das tecnologias da informação e da comunicação no Brasil 2010. São Paulo: Comitê Gestor da Internet no Brasil, Chan, Y.E. & Huff, S.L. (1993) Strategic information systems alignment. Business Quarterly, 58, pp. 51- 55. Colângelo, L., Filho (2001) Implantação de sistemas ERP.- Um enfoque de longo prazo. São Paulo: Atlas. Corrêa, H.L. (1995) Aspectos a considerar na seleção de uma solução ERP para médias empresas. Disponível em: <http://www.correa.com.br/biblioteca/artigos/A04_Computerworld_artigo_escolha_softwa re.pdf>. Acesso em: 20 maio 2003. Coutinho, C.P. & Chaves, J.H. (2002) O estudo de caso na investigação em tecnologia educativa em Portugal. Revista Portuguesa de Educação, 15, pp. 221-243. Dickson, G.W. & Nechis, M. (1984) Key information system issues for the 1980’s. MIS Quartely, 8, pp. 135-159. Esteves-Sousa, J. & Pastor-Collado, J. (2000) Towards the unification of critical success factors for ERP implementations. In: 10th Annual Business Information Technology (BIT) 2000 Conference, 2000. Manchester. Fitzgerald, G. (1998) Evaluating information systems projects: a multidimensional approach. Journal of Information Technology, 13, pp. 15-27. Ganly, D. (2008, August) Address Five Key Factors for Successful ERP Implementations. Gartner Research. ID Number: G00160011 Hair, J.F., Jr, Babin, B., Money, A.H. & Samouel, P. (2005) Fundamentos de métodos de pesquisa em administração. Porto Alegre: Bookman, Henderson, J.C. & Venkatraman, N. (1993) Strategic alignment: Leveraging information technology for transforming organizations. IBM Systems Journal, 32, pp. 4-16. Illa, X.B., Franch, X. & Pastor, J.A. (2000) Formalizing ERP selection criteria. In: 10TH International Workshop on Software Specification and Design, IEEE, San Diego. Koch, C. (1999) The ABCs of ERP - Getting started with Enterprise Resource Planning. CIO Magazine, December, 22. Disponível em <http://teaching.fec.anu.edu.au/INFS3024/Lecture%20Notes/The%20ABCs%20of%20ER P%20-%20Enterprise%20-%20CIOb.pdf>. Acesso em 14 out. 2010.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

296 Medeiros Jr., A. de, Perez, G., Lex, S.

Kumar, V., Maheshwari, B. & Kumar, U. (2002) Enterprise resource planning systems adoption process: a survey of Canadian organizations. International Journal of Production Research, 40, pp. 509-523. Laudon, J.P. & Laudon, K.C. (2011) Sistemas de informações gerenciais. 9 ed. São Paulo: Pearson. Lunardi, G. L., Becker, J.L. & Maçada, A. C. G. (2003) Relacionamento entre Investimentos em Tecnologia de Informação (TI) e Desempenho Organizacional: um Estudo Crosscountry envolvendo os Bancos Brasileiros, Argentinos e Chilenos. Congresso ENANPAD, 2003. 1 CD-ROM McFarlan, F.W. & McKenney J.L. (1982, Sept./Oct.) The information archipelago - Maps and bridges. Harvard Business Review. 60, pp. 109-119. Mendes, J.V. & Escrivão, E., Filho (2007, maio-ago) Atualização tecnológica em pequenas e médias empresas: proposta de roteiro para aquisição de sistemas integrados de gestão (ERP). Gestão e Produção. 14, pp. 281-293. O'Brien, J.A. & Marakas, G.M. (2013) Administração de Sistemas de Informação. Uma introdução. 15. ed., Porto Alegre: McGrawHill/Bookman. Perez, G. & Zwicker, R. (2005) Fornecedores de Serviços de Tecnologia da Informação: Estratégias de Seleção. CATI - Congresso Anual de Tecnologia de Informação, São Paulo. Perez, G. (2007) Adoção de inovações tecnológicas: um estudo sobre o uso de sistemas de informação na área de saúde. 2007. 180 p + apêndices e anexos. Tese (Doutorado) – Departamento de Administração, Faculdade de Economia, Administração e Contabilidade, Universidade de São Paulo, São Paulo. Presley, A. & Meade, L. (1999) Strategic alignment and IT investment selection using the Analytic Network Process. In: Americas Conference on Information Systems (ACIS'99), Milwaukee. Rockart, J.F. (1979, Mar./Apr) Chief executives define their own data needs. Harvard Business Review. 57, pp. 81-92. Saaty, T. L. (2000) Fundamentals of decision making and priority theory: with the Analytic Hierarchy Process. Pittsburgh: RWS Publications. Saaty, T. L. (2001) Decision making with dependence and feedback. The analytic network process. 2. ed. Pittsburgh: RWS Publications. Saaty, T. L. (2005) Theory and applications of the analytic network process. Decision making with benefits, opportunities, costs, and risks. Pittsburgh: RWS Publications. SHIMIZU, T. (2010) Decisão nas organizações: Introdução aos problemas de decisão encontrados nas organizações e nos sistemas de apoio à decisão. 5. ed. São Paulo: Atlas. Teltumbde, A. (2000) A framework for evaluating ERP projects. International Journal of Production Research, 38, pp.4507- 4520. Turban, E.; Leidner, D.; McLean, E., & Wetherbe, J. (2010) Tecnologia da Informação para Gestão - Transformando os Negócios na Economia Digital. Porto Alegre: Bookman. Turoff, M. & Linstone, H.A. (1975) The Delphi Method. New York: Addison Wesley Publishing Company Inc. Wright, J.T.C. (1985, novembro) A Técnica Delphi: Uma ferramenta útil ao planejamento do Brasil? In: III Encontro Brasileiro de Planejamento Empresarial São Paulo. SPE – Sociedade Brasileira de Planejamento Empresarial, pp. 199-207. Yin, R.K. (2010) Estudo de caso: planejamento e métodos. 4. ed. Porto Alegre: Bookman.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.277-296

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 297-326 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200005

INFORMATION TECHNOLOGY GOVERNANCE IN PUBLIC ORGANIZATIONS: HOW PERCEIVED EFFECTIVENESS RELATES TO THREE CLASSICAL MECHANISMS GOVERNANÇA DE TI EM ORGANIZAÇÕES PÚBLICAS: COMO A EFETIVIDADE PERCEBIDA SE RELACIONA COM TRÊS MECANISMOS CLÁSSICOS Gelson Heindrickson Tribunal de Contas da União, Brasilia, DF, Brazil Carlos D. Santos Jr. University of Brasilia, Brasilia, DF, Brazil _____________________________________________________________________ ABSTRACT Information technology (IT) governance has received a lot of attention lately, with a growing strategic importance currently being given to IT by both public and private organizations. This justifies the existence of a body of scientific literature on IT governance, to which this paper belongs and makes an attempt to contribute to. Towards that end, an empirical study was performed involving 57 public organizations of the Brazilian federal administration, examining the relationship between three governance mechanisms – IT steering committee, IT solution manager, and IT investment portfolio management process and IT governance effectiveness. Based on the literature review, a conceptual model was developed to express the causal relations that these constructs were expected to hold with one another. Through a customdesigned questionnaire submitted to over 180 federal public employees, the causal model was tested using mediation analysis and mostly confirmed. Results indicate that Portfolio Management should always be taken into account for analyses that aim to evaluate the effects of IT steering committees and solution managers on IT governance effectiveness. This means that a nonexistent or an underperforming Portfolio Management Process can lead to a reduction or cancellation of the potential positive _____________________________________________________________________________________________ Manuscript first received/Recebido em: 13/04/2013 Manuscript accepted/Aprovado em: 26/06/2014 Address for correspondence / Endereço para correspondência

Gelson Heindrickson, Tribunal de Contas da União, Assessoria de Segurança da Informação e Governança de TI E-mail: gelsonh@tcu.gov.br Carlos D Santos Jr., PhD, Laboratório de Estudos Avançados sobre Dados Abertos e Software Livre,Programa de Pós-graduação em Administração (PPGA) Dep. de Administração (ADM) Universidade de Brasilia (UnB) E-mail: carlosdenner@unb.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

298 Heindrickson, G., Santos Jr., C. D.

contributions of the other two mechanisms to IT governance. By informing decision makers and public managers at some of the main federal public organizations in the country on how to plan and deploy IT to promote a more effective governance, the conclusions presented herein fill a previous knowledge gap in the complementarity and the joint effectiveness of three IT governance mechanisms on the IT dynamics of key public organizations.

Keywords: IT governance; Public organizations; Public administration; Effectiveness; Governance mechanisms; Information Technology; Mediation analysis; Governance. RESUMO A Governança de TI é uma área de estudos recente que tem despertado muito interesse devido à importância cada vez maior da Tecnologia da Informação (TI) para as organizações, sejam elas públicas ou privadas. Isso justifica a existência de uma literatura científica sobre o tema governança de TI na qual este estudo se enquadra e para cujo desenvolvimento busca contribuir. Para tanto, realizou-se pesquisa empírica envolvendo 57 organizações públicas integrantes do governo federal brasileiro na qual foram examinadas as relações entre três mecanismos de governança – Comitê de TI, Gestores da Solução de TI e Processo de Portfólio de Investimento em TI – e a Efetividade da Governança de TI. Com base na literatura revisada, um modelo conceitual foi proposto para expressar as relações que esses construtos deveriam ter uns com os outros. Um questionário foi desenvolvido com base nesse modelo e aplicado a mais de 180 servidores públicos federais. As relações Causais foram testadas com análise de mediação e detectadas, em sua maioria, de acordo com o modelo. Os resultados indicam que o desempenho do Processo de Portfólio deveria sempre ser considerado em análises que tenham por objetivo avaliar os efeitos de Comitê de TI e de Gestores da Solução na efetividade da governança de TI. Isso significa que a não existência ou o baixo desempenho do Processo de Portfólio pode reduzir ou anular a contribuição positiva dos outros dois mecanismos para a efetividade da governança de TI. Por informar tomadores de decisão e gestores dos resultados em algumas das principais organizações da administração federal brasileira no planejamento e utilização TI em busca de efetividade da governança, as conclusões apresentadas neste artigo preenchem um vazio de conhecimento sobre as complementaridades e os efeitos de três mecanismos de governança na dinâmica dessas organizações. Palavras-chave: Governança de TI; Organizações públicas; Administração pública; Efetividade; Mecanismos de governança; Tecnologia da Informação; Análise de mediação; Governança.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

299

1. INTRODUCTION

Information Technology (IT) is currently considered a critical and strategic asset for organizations, both public and private (Affeldt & Vanti, 2009; Albertin & Albertin, 2008a; Albertin & Albertin, 2008b; Assis, 2011; ISACA, 2012). In order for an institution to obtain the benefits expected from IT use, at acceptable levels of risk and cost, IT governance must be established and maintained (ABNT, 2009; Assis, 2011; Machado, 2007; Ramos, 2009). IT governance can be understood as a set of policies, organizational structures, work processes, roles and responsibilities that are established by the top management in order to steer IT actions and exert control over the use and management of IT throughout the institution (Mello, 2006; Mendonça, 2013; ISACA, 2012). Studies carried out at the international level by Weill & Ross (2006) and at the national level by Lunardi, Becker & Maçada (2012) reveal that companies that had implemented IT governance performed better in comparison to those that did not or those with a deficient implementation. An IT governance mechanism that is often mentioned in literature and recommended by audit entities such as TCU – the Brazilian SAI (Supreme Audit Institution) – is the IT Steering Committee or IT Executive Committee (BRASIL, 2008). Surprisingly, a study carried out by Ali & Green (2012) has not identified statistically significant relationships between the action of the IT steering committee and the effectiveness of IT governance, reaching a paradoxical conclusion. That study, though gave priority to private companies and the sample was intentionally filtered in order to include only institutions with a certain level of indirect execution (outsourcing) of IT activities. Therefore, it would not be appropriate to generalize the conclusions of that study to all situations. Further investigations are required. Additionally, it must be taken into account that the respondents to the Ali & Green (2012) survey were representatives of only one group of stakeholders of enterprise IT: IT auditors. This apparent contradiction provided the initial motivation for the present study as it pointed out the need for investigation that would take into account the specificities of public institutions and engage other stakeholders. Another motivation was the opportunity to include the observation of two IT governance mechanisms in the analysis, which, according to the theoretical framework surveyed, would be closely related to the actions of the IT Executive Committee, namely: the IT Investment Portfolio Management Process and the IT Solution Manager. Another aspect that has raised our interest in carrying out the present research was the lack of materials available in terms of best practice models, scientific papers and publications in general specifically addressing IT governance in public institutions. Hence, based on widely disseminated assumptions regarding good practices in IT governance, the aim of the present study is to empirically investigate the relationships that exist between the following IT governance mechanisms at federal

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

300 Heindrickson, G., Santos Jr., C. D.

public institutions: IT Investment Portfolio Management Process, IT Steering Committee and IT Solution Manager. Accordingly, our research question is: “How does the action of the IT Committee and of IT Solution Managers relate to the performance of the IT Investment Portfolio Management Process and how do these three mechanisms interact with the effectiveness of IT Governance?”. It should be highlighted that only federal public institutions have been included in this research and that we did not aim for an institutional perspective, but for the personal perception of public employees representing the multiple stakeholders of the IT organization. This paper provides empirical evidence that sheds new light on the relationships that exist between the IT governance mechanisms investigated. It can support public institutions in their implementation of IT governance, as well as provide input for recommendations issued by auditing authorities to the entities under their jurisdiction. Below, is a brief theoretical framework on the effectiveness of IT governance and the governance mechanisms that have been evaluated and presented, followed by a description of the methods used in the research. Later, we present the results obtained and discuss our conclusions and final remarks. 2. THEORETICAL FRAMEWORK 2.1 IT governance effectiveness Effective IT governance contributes to improved IT performance, which, on its turn, contributes to improved organizational performance (Assis, 2011; Machado, 2007; Mendonca et al., 2013; Weill & Ross, 2006). But, one might ask, what is an improved organizational performance? A quite objective answer is provided in Cobit 5, an important IT governance framework, that associates IT performance and organizational performance via the creation of value for the business and clarifies that “creating value is to realize benefits at optimal resource cost whilst optimizing risk” (ISACA, 2012, p. 17). Value creation can also be described as the achievement of certain IT objectives related to certain generic corporate goals, applicable to all organizations (ISACA, 2012, p. 18, Figure 4). These objectives aim to ensure the following aspects, among others: that planned actions and those under way in the IT departments be aligned with the organization’s business strategy, so as to give priority to the most important requests of the business to the IT department; that meeting these requests should be in compliance with the requirements, deadlines, quality and the costs agreed; that costs and risks of IT initiatives should be managed and that the expected benefits of the IT actions carried out should be achieved (ISACA, 2012b). Along the same lines, Dolci & Maçada (2011) have identified a comprehensive set of benefits targeted by organizations regarding different dimensions of IT investments. As a higher degree of fulfillment of objectives leads to the creation of greater value – which implies in a greater effectiveness of IT governance – this causal relationship justifies the use of the aspects addressed by the objectives mentioned in the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

301

questions of the survey form that evaluated the effectiveness of IT governance (Appendix A, item 4). 2.2 IT investment portfolio management Portfolio management has become a popular topic following the publication of Markowitz (1952), aimed at financial markets. McFarlan (1981) and Ward (1990) have pioneered the use of the portfolio approach in selecting IT investments. Studies carried out by Weill, Woerner & Rubin (2008) have followed up on the evolution of the use of the IT governance concepts and practices by large private international corporations. In the national scenario, Moraes & Laurindo (2003) have shown positive results obtained from the implementation of the IT project portfolio management process. Dolci & Maรงada (2011), was based on case studies carried out with national companies with high investments in information technology, have identified the most important aspects to be considered by the four dimensions usually taken into account regarding IT investment portfolios. Cobit 5 (ISACA, 2012b) includes IT investment portfolio management among decision-making tools and for supporting the monitoring of actions associated to IT management and use. In this model, the topic is addressed by the following processes and practices: a) In the domain of IT governance: EDM02.01 Evaluate value optimisation, EDM02.02 Direct value optimization and EDM02.03 Monitor value optimisation; b) In the domain of IT management: APO05.01 Establish the target investment mix, APO05.02 Determine the availability and sources of funds, APO05.03 Evaluate and select programmes to fund, APO05.04 Monitor, optimise and report on investment portfolio performance, APO05.05 Maintain portfolios and APO05.06 Manage benefits achievement. In the ValIT framework (ITGI, 2008), IT investment portfolio management is addressed under the Portfolio Management (PM) domain. Its objective is to guarantee that organizations may obtain optimal value for its IT investments. Practices associated with portfolio evaluation and balancing fall within this domain, as well as the definition of criteria for these activities. ValIT also addresses the follow up of the portfolios overall performance. Valuable information sources on how to establish or improve the IT investment portfolio process are: the ITIM framework, designed and published by the United States General Accounting Office (GAO, 2004) and the study carried out by Weill, Woerner & Mcdonald (2009). The standard defined under The Standard for Portfolio Management (PMI, 2008) and the model proposed by Archer & Ghasemzadeh (1999) differentiate themselves in one aspect: they are not limited to the management of IT portfolios, but are also applicable to the management of institutional program and project portfolios in general. As described by the above-mentioned sources, the objective of the IT investment portfolio management process is to optimize the use of institutional resources, so as to select a set of projects and programs capable of providing the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

302 Heindrickson, G., Santos Jr., C. D.

greatest possible returns to the organization. Selected projects and programs must be aligned with corporate strategies and, according to the methodology, their risks, costs, quality, deadlines and levels of service must be properly managed. It has been said that, in order to meet its objectives, a portfolio management process requires the definition of decision-making structures and the fulfillment of some common stages or phases. Despite the fact that each model uses slightly different names, the main ones are: identification and analysis of components/projects, selection and prioritization of components, authorization, monitoring, reviews and risk management. Requirements for portfolio management are project management and program management, but are not limited to them. This set of characteristics described in literature has provided input for the phrasing of the survey questions that evaluated the performance of the governance mechanism of the IT Investment Portfolio Process (Appendix A, item 1). The objective and the characteristics of the portfolio management process are clearly congruent with the objectives of the so-called focus areas of IT Governance: strategic alignment, value delivery, resource management, risk management and performance measurement (ITGI, 2007). Hence, it is reasonable to consider the following hypothesis: improvements in the performance of the IT investment portfolio process have a positive influence on the effectiveness of IT governance (hypotheses H3 in Figure 1). 2.3 IT solution manager There are several different names for the “IT Solution Manager” governance mechanism: business executive, business area, project or investment sponsor or business process owner. In the present article we have chosen to use “IT Solution Manager”, as it can be applied to the whole life cycle of an IT solution; it also is less restrictive, as IT may support not only business processes in the strict sense of the word, but also administrative processes, support processes and even processes of the IT department itself. The role of the TCU IT Solution Manager is internally regulated by TCU Administrative Order no. 156 of 2012 (BRASIL, 2012), whereas other public entities have their own norms on that issue. It should be noticed that, even though restricted to situations involving the provision of contract-based IT solutions, Normative Instruction no. 4/2010 of the Secretariat for Logistics and Information Technology of the Ministry of Planning, Budgeting and Management (SLTI), the central authority of the System for the Administration of Information Resources and Information Technology of the Federal Executive Power (SISP), defines some attributions of the IT Solution Manager role in the items that address the responsibilities of the contracting planning team (BRASIL, 2010). Within the ValIT framework (ITGI, 2008), the relevance of the IT solution manager role is expressed in the Investment Management (IM) domain of that model. Its objective is to guarantee that individual portfolio components may contribute to generating value for the organization. This domain includes practices related to the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

303

identification of business requirements, expected benefits, as well as the preparation of the Business Case, a document that issues a formal request and registers essential information about the initiative. It also addresses the follow up of individual performance, i.e., monitors the fulfillment of the benefits of the program or project throughout their life cycle. In Cobit 5 (ISACA, 2012b), the responsibilities of the IT Solution Manager are presented under different items of the framework. The “RACI” tables show the responsibilities attributed to the Business Executive or the Business Process Owner, i.e., to the unit or sector benefitting from the investments in the IT solution. That can be noticed, especially, in the description of the following processes, key practices and activities of Cobit 5: BAI01 Manage Programmes and Projects (several key practices), BAI02 Manage Requirements Definition (several key practices), APO12 Manage Risk, APO05.06 Manage benefits achievement, APO09.03 Define and prepare service agreements and PO09.04 Monitor and report service levels. According to the IT Governance Institute, an IT Solution Manager is responsible for: acting as the sponsor of the IT solution, defining business requirements for the IT solution; defining and controlling levels of service for the IT solution; continuously evaluating the benefits of the IT solution; evaluating the risks to the business that are associated with the IT solution; providing resources and establishing priorities for the IT solution, among other responsibilities (ITGI, 2003, p. 51). Peppard, Ward & Daniel (2007) have revealed that the value of IT investments can only be obtained by means of an ongoing identification and management of the benefits achieved throughout the whole life cycle of the IT solution. And this is the responsibility of business managers, i.e., this responsibility lies with those who we have called “IT Solution Manager”. For the purposes of this study we have considered that the IT Solution Manager is the organizational unit with the greatest interest in the investment, development or hiring of an IT solution (be it a system, software, app or service provided by the IT department). It should be noted that, even though the operational activities of the IT Solution Manager can be delegated to sub-units, departments or specialists, this is not relevant to the analysis proposed in this study. The questions of the survey that evaluated the performance of the IT Solution Manager governance mechanism were based on the set of characteristics described in literature (see Appendix A, item 3). The responsibilities of the IT Solution Manager described herein can be easily related to the common roles found in portfolio process models, such as the sponsor and program manager roles (PMI, 2008), including activities that are considered necessary at certain stages or phases of a portfolio management process, especially during the identification, authorization, review and report, and risk management stages (PMI, 2008), as well as during the individual project analysis, project development and phasegate evaluation phases (Archer & Ghasemzadeh, 1999). Therefore, the following hypothesis is to be considered: improvements in the actions of IT solution managers have a direct and positive influence on the performance of the IT investment portfolio process (hypothesis H2 in Figure 1). Likewise, this simple logical and causal

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

304 Heindrickson, G., Santos Jr., C. D.

relationship seems to indicate that several attributions of the IT Solution Manager can contribute to the fulfillment of the following objectives of IT Governance focus areas: value delivery, resource management, risk management and performance measurement (ITGI, 2007). Therefore, it is reasonable to consider the following hypothesis as well: improvements in the performance of IT solution managers have a direct and positive influence on the effectiveness of IT governance (hypothesis H5 in Figure 1). 2.4 IT Steering Committee Weill & Ross (2006) have evaluated large numbers of large international corporations over several years and report that IT Steering Committees are important governance mechanisms used by top performing companies. Castro & Carvalho (2010) also have identified the need for a committee composed by representatives of several different areas of the organization to take decisions regarding the selection and prioritization of projects. Likewise, a case study that has been evaluated by Moraes & Laurindo (2003) shows positive results deriving from the joint participation of managers from IT and business areas in the decision-making associated to the selection and prioritization of projects. The organization is responsible for establishing an IT Steering Committee, composed of directors and managers from the IT and business areas. Among other responsibilities, as established in Cobit 5 under APO01.01 Define the organizational structure key practice, that the committee should: “determine prioritisation of ITenabled investment programmes in line with the enterprises business strategy and priorities; track status of projects and resolve resource conflicts; and monitor service levels and service improvements” (ISACA, 2012b). Cobit 5 (ISACA, 2012b) also presents other attributions of the IT Steering Committee in its “RACI” tables. That can be especially noticed in the descriptions of the following processes, key practices and activities: BAI01 Manage Programmes and Projects (several key practices), APO12 Manage Risk, and APO05.06 Manage benefits achievement. According to the IT Governance Institute, it is the responsibility of the IT Steering Committee to: participate in the approval of new IT solutions; evaluate the alignment of proposals for new IT solutions with the organizational strategies; define priorities for projects; ensure that all costs and benefits of the proposals for new IT solutions have been identified; guarantee that projects are being risk-managed; follow up the progress of relevant IT projects, among other responsibilities (ITGI, 2003, p. 52). According to TCU, all public organizations, in addition to other actions, must establish an IT steering committee, “in order to provide for the allocation of public resources in accordance with the organization’s needs and priorities” (BRASIL, 2008). In compliance with its own recommendations, the TCU has implemented a governance mechanism, as part of the Court’s internal processes, for that end, called IT Management Committee. The responsibilities of the Committee are established under TCU Resolution no. 247/2011 (BRASIL, 2011) and TCU Administrative Order no. 156/2012 (BRASIL, 2012).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

305

In line with the best IT governance practices and the recommendations of the audit entities, the SISP Information Technology General Strategy (Estratégia Geral de Tecnologia da Informação – EGTI) for the 2011-2012 period also foresees the establishment of IT Committees by the organizations that are part of the system (BRASIL, 2011b). The IT Committee that was evaluated by this study holds the characteristics and attributions of the IT steering committee described in the sources mentioned above, having thus set the basis for the drafting of questions for the survey questionnaire which have evaluated the governance mechanism (Appendix A, item 2). It can be easily noticed that the attributions of the IT steering committee described herein are related to the usual roles found in portfolio process models, such as the Portfolio Review Board and the Portfolio Manager roles (PMI, 2008). Their activities are considered necessary during the stages or phases of a portfolio management process, especially during the prioritize, balance, authorize, review and report and risk management stages (PMI, 2008), or during the optimal portfolio selection and phase-gate evaluation (Archer and Ghasemzadeh, 1999) phases. Hence, the evaluation of the following hypothesis is applicable: improvements in the action of the IT committee have a direct and positive influence on the performance of the IT investment portfolio (hypothesis H1 in Figure 1). Additionally, it can be noticed that there are intersections between the attributions of the IT Steering Committee and the following objectives of the IT Governance focus areas: strategic alignment, value delivery and risk management (ITGI, 2007). Therefore, it is reasonable to consider the following hypothesis as well: improvements in the performance of the IT Steering Committee have a direct and positive influence on the effectiveness of IT governance (hypothesis H4 in Figure 1). 2.5 Conceptual model This study intends to investigate the conceptual model presented in Figure 1. The variables analyzed – which are often mentioned in several parts of this document – match the identifiers shown in brackets in each rectangle of the figure: COMITE, GESTSOL, PROCPORTF and EFETGOVTI.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

306 Heindrickson, G., Santos Jr., C. D.

Figure 1 â&#x20AC;&#x201C; Conceptual model and hypotheses

The hypotheses that will be evaluated in this study, concerning the relationships between governance mechanisms and between these and IT governance effectiveness, are represented by arrows and identifiers in Figure 1, namely: a) H1: Improved action by the IT steering committee has a direct and positive influence on the performance of the IT investment portfolio process. b) H2: Improved action by IT solution managers has a direct and positive influence on the performance of the IT investment portfolio process. c) H3: Improved performance of the IT investment portfolio process has a direct and positive influence on IT governance effectiveness. d) H4: Improved action by the IT steering committee has a direct and positive influence on IT governance effectiveness. e) H5: Improved action by the IT solution managers has a direct and positive influence on IT governance effectiveness. f) H6: The overall effect of improvements in IT steering committee action is positive on IT governance effectiveness, i.e., the balance between the direct and indirect effects, via portfolio process, of IT committee action on IT governance effectiveness is a positive one. g) H7: The overall effect of improvements in IT solution manager action is positive on IT governance effectiveness, i.e., the balance between the direct and indirect effects, via portfolio process, of IT solution manager action on IT governance effectiveness is a positive one.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

307

3. METHODS The survey was done through a questionnaire made available over the Internet between October 10th, 2012 and October 15th , 2012. Federal public servants made up the target audience, encompassing the following IT stakeholders in the participating organizations: internal IT clients, IT servants, internal auditors and IT consultants, encompassing managers and non-managers. The survey was distributed via email to the representatives of the TI Controle community, which gathers several IT managers from federal public institutions. SLTI sent the same survey to public servants that had subscribed to the entity’s portal as well. Additionally, the survey organizer spread it to its contacts and provided a website to facilitate the collection of information and access to the form by interested respondents. The four questions of the questionnaire that evaluated IT Steering Committee action (COMITE), the five questions that evaluated the IT Solution Manager action (GESTSOL) and the five questions that evaluated the IT investment portfolio process (PROCPORTF) had been drafted based on the main characteristics of these governance mechanisms, as described in academic studies, best practice models and other literature sources reviewed. All questions related to independent variables and mediator (COMITE, GESTSOL and PROCPORTF) had been measured according a 5-point Likert scale, which evaluated two dimensions for each question: i) the relevance of the question for IT governance effectiveness, as perceived by the respondent; ii) the degree to which the action or activity proposed by the question had been fulfilled by the institution, according to the respondent’s perception. Figure 2 is an example of how these two dimensions evaluated for one of the questions addressing the COMITE variable.

Figure 2 – Sample question for the evaluation of the “relevance” and “fulfillment” dimensions

The dependent variable – IT governance effectiveness (EFETGOVTI) – was evaluated through a proxy: the respondents’ satisfaction regarding the achievement of objectives and the realization of the benefits that are expected from good IT governance and management, aiming at the creation of value for the organization, as stated in the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

308 Heindrickson, G., Santos Jr., C. D.

literature review section. The six questions regarding this variable were also evaluated based on a 5-point Likert scale. For each variable analyzed in the survey, Chart 1 summarizes the main reference sources on the topic and the questionnaire items (Appendix A) that represent the questions used for measuring these variables.

Variable

Questionnaire Questions

Main Bibliographic References

PROCPORTF

MCFARLAN, 1981; WARD, 1990; ARCHER & GHASEMZADEH, 1999; GAO, 2004; ITGI, 2008; PMI, 2008; WEILL, WOERNER & MCDONALD, 2009; ISACA, 2012b. ITGI, 2003, p. 52; MORAES & LAURINDO, 2003; WEILL & ROSS, 2006; BRASIL, 2008; CASTRO & CARVALHO, 2010; BRASIL, 2011; BRASIL, 2011b; BRASIL, 2012; ISACA, 2012b; ALI & GREEN, 2012.

COMITE

GESTSOL

EFETGOVTI

ITGI, 2003, p. 51; ITGI, 2007; ITGI, 2008; PEPPARD, WARD & DANIEL, 2007; BRASIL, 2010; ISACA, 2012b; BRASIL, 2012.

WEILL & ROSS, 2006; DOLCI & MAÇADA, 2011, Figure 2, p. 363; ISACA, 2012, p. 18, Figure 4; ISACA, 2012b, p. 226, Figure 17.

(Appendix A)

Cronbach’s Alpha Coefficient

Items 1.1 to 1.5

0.7938

Items 2.1 to 2.4

0.8523

Items 3.1 to 3.5

0.7841

Items 4.1 to 4.6

0.8273

Chart 1 – Variables, bibliographic references and questionnaire questions For the independent variables and the mediator variable (COMITE, GESTSOL and PROCPORTF), their measurement in each answer was calculated as follows: for each question that is part of the variable, the average value obtained from the answers of all respondents to the “question relevance” dimension was used as a weight that was applied to each answer to the “degree of fulfillment” of the question. The weighted average of the set of questions of a variable times its weights, converted into a 10-point scale, generated a final average of this variable in each answer. The strategy for calculating the measure of a variable based on its components is based on the stated preference weights approach (Decancq & Lugo, 2010, p. 17), according to which the relevance/weight is defined by the respondents themselves. For the dependent variable (EFETGOVTI), the average measure for each answer was calculated based on the simple average of values to the answers to their questions, converted into a 10-point scale. This strategy for calculating the measure of a variable based on its components was based on the expert opinion weights approach

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

309

(Decancq & Lugo, 2010, P. 16); all sources mentioned in the theoretical framework were considered experts – with an equivalent relevance – which provided the basis for the selection of questions. The selection of items used for measuring the variables is fully justified as it is based on the opinions of respondents and of experts on the topic, according to the previously mentioned approaches. Nevertheless, we also carried out a statistical analysis of the internal consistence or reliability of the questionnaire items that make up each variable of the study. Chart 1 shows that the values of Cronbach’s Alpha coefficient obtained for each variable fall within the acceptable range, that is, between 0.70 and 0.95 (Tavakol & Dennick, 2011). It also has been found that the eventual exclusion of items would not improve the Alpha coefficient value in none of the variables, which reinforces the relevance that is given to the items by literature. The model presented in Figure 1 shows that the IT Steering Committee and the IT Solution Manager can have a direct effect on IT governance effectiveness, as well as indirect and total effects through the Portfolio Process. To evaluate the direct effects shown in this figure, we used the multiple regression analysis technique. In order to take into account the indirect and total effects, the mediation analysis statistics method was used in the study. Its aim is to test the relationships observed between a set of independent variables and a dependent variable upon the inclusion of an additional variable, known as mediator. In the mediated model, the hypothesis is that the independent variable influences the mediator variable, which, on its turn, influences the dependent variable. To be highlighted is that the models that can be analyzed with the method are not limited to three variables and can be applied to several models, with any number of dependent variables and mediator variables. The importance of this type of analysis has achieved notoriety following the studies by Baron and Kenny (1986), and very effective modern techniques are currently available, such as those mentioned by MacKinnon (2008) and Hayes (2009). We used the “R” statistical software to calculate Cronbach’s Alpha coefficient and carry out the multiple regression analysis, as well as the auxiliary correlation tests used in this study. For the mediation analysis we have used the SPSS ® Statistics software and the “MEDIATE” (Hayes, 2012a) and “PROCESS” (Hayes, 2012b) macros, which are support tools available on the internet that can be accessed and used by the general public. 4. RESULTS Below we present the characteristics of the sample and the results of the tests described in the previous section. To test the hypotheses shown in Figure 1, we initially explore the existing correlations between the variables and evaluated the direct effects by means of a multiple regression test. Next, in order to obtain the final results, we used the mediation test.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

310 Heindrickson, G., Santos Jr., C. D.

4.1 Sample characteristics

Table 1 â&#x20AC;&#x201C; Sample characteristics

Table 1 presents the distribution of respondents with regards to important grouping characteristics. Other peculiarities of the sample can be found in Appendix B. We had a total of 189 respondents, associated to 57 different institutions, and about one third of all respondents were TCU employees. The large number of TCU respondents can be explained by the fact that the research organizer works at TCU, which has facilitated communication and access to people. An analysis of the rate of respondents in relation to the requests made is not applicable as the main means of divulgation outside TCU were not under the control of the research organizer (indirect means: divulgation made by the IT Control Community and by SISP to its members). 4.2 Preliminary Analysis The correlation test calculated the Pearson product-moment for each pair of variables and the multiple regression test allowed to estimate coefficients for the calculation of PROCPORTF based on COMITE and GESTSOL and for the calculation

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

311

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

of EFETGOVTI based on these three variables. The results are summarized in Table 2 and Table 3, respectively.

Variable

Average Standard Deviation

Correlations COMITE

GESTSOL

PROCPORTF

EFETGOVTI

COMITE

6.33

2.08

GESTSOL

6.11

1.58

0.38***

PROCPORTF

6.03

1.59

0.41***

0.48***

EFETGOVTI

5.64

1.35

0.35***

0.40***

0.58***

Obs.: N=189; ***p< 0.001; ** p<0.010; * p<0.050; ^p<0.100

Table 2 – Averages, standard deviations and correlations between variables As shown in Table 2, initial results indicate there is a statistically significant correlation between all variables of the model, which could support the validity of some of the hypotheses presented in Figure 1.

D i r e c t

e f f e c t s

Coefficients to

Variable

calculate PROCPORTF

calculate EFETGOVTI

COMITE

0.21*** (H1)

0.07

GESTSOL

0.37*** (H2)

0.11^ (H5)

PROCPORTF

0.41*** (H3)

(H4)

Obs. Obs.: N=189; ***p< 0.001; ** p<0.010; * p<0.050; ^p<0.100

Table 3 – Results of the regression tests, with estimates for the direct effects of COMITE and GESTSOL over PROCPORTF and of these three variables over EFETGOVTI Additionally, the results of the regression test, shown in Table 3, indicate that the direct effects of GESTSOL and COMITE over PROCPORTF are substantial and statistically significant, which allows us to accept hypothesis H1 and H2 of Figure 1. The direct effects of PROCPORTF over EFETGOVTI also have a high value and are statistically significant, which confirms hypothesis H3 of Figure 1. The direct effects of GESTSOL and COMITE over EFETGOVTI evaluated by the regression test, though, demonstrated to be small and are not statistically

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

312 Heindrickson, G., Santos Jr., C. D.

significant, with confidence levels lower than 95% and 90%, respectively, which leads us to reject hypotheses H4 and H5 of Figure 1. 4.3 Mediation analysis According to the mediation analysis theoretical framework, the direct effect is calculated the same way as in the regression analysis, whereas the coefficient that measures the value of an indirect effect is given by the product of the coefficients of the mediated direct effects. The total effect, though, is calculated as the sum of the direct and indirect effects that have been measured between the same variables. Additionally, data must be submitted to hypothesis tests that allow the validation of the estimated values for such effects, that is, if they can be considered statistically significant (Baron & Kenny, 1986; Mackinnon, 2008; Hayes, 2009). As seen, the preliminary regression analysis has rejected the direct effects of COMITE (H4) and GESTSOL (H5) on EFETGOVTI, which could discourage the performance of tests regarding the proof of total effects (H6 and H7). Nevertheless, it is known that a variable can have a positive total effect on another variable even though a direct effect is not identified (HAYES, 2012b). This is because its effect may occur through a third variable, a mediator. In fact, the mediation analysis performed afterwards with the support of the “MEDIATE” macro (Hayes, 2012a) revealed statistically significant indirect effects and total effects, at a confidence level higher than 95% of the GESTSOL and COMITE variables on EFETGOVTI variables when considering the PROCPORTF variable as a mediator. In view of the results of the mediation analysis shown in Table 4, hypothesis H6 and H7 of Figure 1 can be accepted regarding the total effects of COMITE and GESTSOL over EFETGOVTI, respectively.

Direct effects

Indirect effects

Total effects

Variable

Over EFETGOVTI

via PROCPORTF

over EFETGOVTI

COMITE

0.07

0.08~

0.15** (H6)

GESTSOL

0.11^

0.15~

0.26*** (H7)

Obs.: N=189; ***p< 0.001; ** p<0.010; * p<0.050; ^p<0.100; ~ i interval LLCIULCI does not contain 0

Table 4 – Mediation test results, with estimates of the direct, indirect and total effects of COMITE and GESTSOL over EFETGOVTI The mediation test was configured to simulate 10,000 samples or bootstrap samples, while the recommendation by Hayes (2009) is to use at least 5,000. It should be underscored that, in mediation tests using the bootstrap technique, the interpretation of the probability or confidence level of the indirect effect is not based on a “p” value, but on the values resulting from the LLCI-ULCI interval: if the interval does not

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

313

contain a zero value, the mediation hypothesis is accepted (Hayes, 2009; Hayes 2012a; Mackinnon, 2008). In view of the high number of TCU respondents in the sample, the mediation test was repeated by excluding the answers of these participants, for comparison with the results obtained from the full sample. Despite the variations in the values of estimated coefficients for the reduced sample, it was found that the calculated direct, indirect and total effects presented similar results and led to statistically significant levels of confidence, such as those obtained from using the full sample. 4.4 Consolidated results Figure 3 summarizes the results obtained from the mediation test: each path or hypothesis presents the estimated regression coefficient, rounded to two decimal places, as well as their statistical significance.

Figure 3 â&#x20AC;&#x201C; Effects calculated through mediation analysis The result of the mediation analysis allows us to discuss how the evaluated IT governance mechanisms relate to each other and with IT governance effectiveness. These relationships and effect sizes, on average, are summarized below on a scale from 0 to 1, which is typical of statistical regression and mediation tests: a) Each point of improved performance in the IT investment portfolio process is equivalent to 0.41 point of improvement on IT governance effectiveness (as per H3 in Figure 3);

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

314 Heindrickson, G., Santos Jr., C. D.

b) Each point of improved IT Steering Committee action is equivalent to 0.15 improvement on IT governance effectiveness (as per H6 in Figure 3); c) Each point of improved IT Solution Manager action is equivalent to 0.26 point improvement on IT governance effectiveness (as per H7 in Figure 3); d) Each point of improved IT Steering Committee action is equivalent to 0.21 point improvement on the performance of the IT investment portfolio process (as per H1 in Figure 3); e) Each point of improved IT Solution Manager action results on a 0.37 point improvement on the performance of the IT investment portfolio process (as per H2 in Figure 3). It should be noted that in the case of COMITE and GESTSOL, this summary has considered the total effects on EFETGOVTI, as the result of the mediation analysis certifies that the effects these mechanisms indirectly exert on IT governance effectiveness, through its influence on the IT investment portfolio, are significant. These results allow us to reach some important conclusions that can contribute to a better understanding of the specific topic being analyzed herein for the evolution of IT Governance as a field of study, and act as practical guidance for public organization leaders, managers and auditors regarding the implementation of the analyzed governance mechanisms, the planning of derived actions and in identifying training needs. Because COMITE (H4) and GESTSOL (H5) have not been found to have statistically significant direct effects on EFETGOVTI, but that statistically significant total effects exist, and considering the mediation of PROCPORTF (H6 and H7), an important conclusion is: eventual efforts to improve the action of the IT Steering Committee and/or of IT Solution Managers depend on the existence of an Investment Portfolio Process in order to have relevant effects on IT governance effectiveness. Finally, it can be concluded that the control entities and best-practice models mentioned in the conceptual sections of the paper are correct to recommend the use of the governance mechanisms analyzed herein, as the present study provides evidence of their contribution to the effectiveness of IT governance. 4.5 Other findings In addition to the conclusions related to the research question and hypotheses presented in the conceptual model shown in Figure 1, other interesting findings, associated to the different group perceptions, are presented in Appendix B. Even though these findings are not directly related to the research question and with the primary objective of the study, we took advantage of the opportunity offered by the collected data and carried out some additional exploratory analyses. The results of which are presented next, aiming at encouraging future complementary research on the topic. In order to carry out these analyses, we used the Microsoft Excel速 software and non-parametric tests provided by the XLSTAT add-in.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

315

With regards to the perceived relevance of the three mechanisms for assessing IT governance effectiveness analyzed in the study (COMITE, GESTSOL, PROCPORTF), the evaluation has led to rather high averages for all groups, with a small standard deviation, as can be seen in Table B1, B2 and B3 of Appendix B. Despite unanimous agreement on the relevance of the mechanisms, there is statistically significant evidence of differences in the perception of the groups, namely: a) Regarding the relevance of the IT Portfolio Process, there is evidence that groups that have taken a course on IT governance and those that have not do have different perceptions. The same difference exists between IT personnel and IT client personnel, as well as between IT auditors/consultants and non-IT auditors/consultants (Table B1); b) Regarding the relevance of the IT Steering Committee, there is evidence of different perceptions between the respondents of other institutions and TCU respondents, as well as between the group that took the IT governance course and the group that did not take the course, as well as between IT auditors/consultants and non-IT auditors/consultants (Table B2); c) With regards to the importance of the IT Solution Manager, there is evidence of different perceptions by the group that took the course on IT governance and the group that did not, and also between IT auditors/consultants and non-IT auditors/consultants (Table B3). There also is evidence of differences in perception between groups regarding the relevance of the eventual causes of non-effectiveness of IT governance that have been evaluated, as shown in Table B4: a) Reasonable difference in perception between TCU respondents and the respondents from other institutions regarding the relevance of Cause 1 – insufficient IT personnel, and also regarding Cause 4 – insufficient financial resources; b) Reasonable difference in perception between IT personnel and IT client personnel regarding the relevance of Cause 1 – insufficient IT personnel, Cause 2 – lack of technical training by IT personnel, Cause 5 – lack of exploitation of alternatives for the provision of IT solutions, and Cause 6 – lack of management training by IT personnel; c) Reasonable difference in perception between IT auditors/consultants and non-IT auditors/consultants regarding the relevance of Cause 4 – lack of financial resources. It is important to note, regarding the results presented in Appendix B, that there is evidence that little difference exists in the perception of the group that exerts a management position and the group that does not carry out this type of activity. This is the case both for the perception about the relevance of the three IT governance mechanisms evaluated (tables B1, B2 and B3), as well as for the perception of the relevance of the six causes of the non-effectiveness of IT governance (table B4) that have been evaluated.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

316 Heindrickson, G., Santos Jr., C. D.

5. CONCLUSIONS The present study analyzed the relationships between three well-known governance mechanisms and the effectiveness of IT governance in Brazilian federal public institutions. Empirical evidence shows that the performance of the IT Steering Committee and of the IT Solution Manager has a positive influence and considerably affects the performance of the IT Investment Portfolio Process. It also shows that these three mechanisms have a positive influence on the effectiveness of IT governance. The influence of the IT Investment Portfolio Process is direct, quite high, statistically significant and easily detected by means of a simple regression analysis. The direct influences of the IT Steering Committee and of the IT Solution Managers on IT governance effectiveness were not proven at statistically significant levels. Nevertheless, the use of a more robust statistical analysis allowed us to detect statistically significant total effects of these two mechanisms on the effectiveness of IT governance, when the IT Investment Portfolio was considered as a mediator. This suggests that, in order to be effective, efforts to improve the performance of the IT Steering Committee or of IT Solution Managers depend on the existence of the IT Investment Portfolio Process. It should be highlighted that the need to use the mediation analysis technique must not be interpreted as some type of weakness in the effects observed: it only indicates that the mechanisms analyzed interact in a more complex way, thus also requiring more sophisticated techniques for the appropriate analysis of their effects. It also should be noted that the results of this study should not necessarily be interpreted as a causal relationship between variables, considering that the specific investigation techniques and statistical methods required for providing such evidence were not used (ideally, controlled experiments). Therefore, the terms “influence” and “effect” used in the study in order to facilitate describing the relationships between the governance mechanisms that were evaluated are better interpreted as a factual reflection of the value variations of a variable in relation to the values presented by another variable, irrespective of the identification of the primary cause of this phenomenon. The performing of this study faced funding and time restrictions that resulted in certain limitations that must be considered when interpreting the results. First, the sample size may be considered limited, given the number of federal public organizations that exist in the country. Second, the disproportional number of respondents per institution may be considered small in some cases and quite large in others, as in the case of TCU, with possible effects on the values estimated by the statistical calculations, a consequence of the different levels of IT governance maturity encountered in the organizations. Third, the fact that the analyzed model did not take into account other IT governance mechanisms, which were intentional, in order to simplify the survey questionnaire and not to discourage respondent participation. However, the absence of other mechanisms implies that other relationships were not evaluated. We suggest that researchers with interest in further deepening the work done in this study may find ways of reducing the aforementioned limitations. On the other hand, this study has innovated in some important aspects, such as taking into account the opinion of the respondents to assess the relevance of the questions, in addition to the measurement of their fulfillment and the application of the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

317

questionnaire to a broad target audience. This audience represented several key IT stakeholders in public organizations, including directors, managers, internal clients and IT technical staff. Remembering that IT governance is the responsibility of higher-level management, normally the board of directors (Weill & Ross, 2006; ABNT, 2009; BRASIL, 2010b; ISACA, 2012), the results of this study can be used as input for decision-making by public organization leaders regarding the implementation of the governance mechanisms analyzed. They may also support the organization’s IT managers and IT governance staff in the identification of training needs and in the planning of communication and educational actions, based on the conclusions presented in Figure 3 – regarding the hypotheses tested based on the conceptual model of Figure 1 – as well as based on the other findings related to the differences in perceptions between groups, as detailed in Appendix B.

REFERENCES ABNT. ABNT NBR ISO/IEC 38500/2009 - Governança corporativa de tecnologia da informação, 2009. Available at: <http://www.abntcatalogo.com.br/norma.aspx?ID=40015>. Accessed on May, 2012. Affeldt, Fabrício S.; Vanti, Adolfo A. Alinhamento estratégico de tecnologia da informação: análise de modelos e propostas para pesquisas futuras. Journal of Information Systems and Technology Management, v. 6, n. 2, p. 203-226, 2009. Albertin, A. L.; Albertin, R. M. M. Tecnologia de Informação e Desempenho Empresarial no gerenciamento de seus projetos: um estudo de caso de uma indústria. Revista de Administração Contemporânea (Printed), v. 12, p. 599-629, 2008a. Albertin, A. L.; Albertin, R. M. M. Benefícios do uso de Tecnologia de Informação para o desempenho empresarial. Revista de Administração Pública (Printed), v. 42, p. 275-302, 2008b. Ali, S.; Green, P. Effective information technology (IT) governance mechanisms: An IT outsourcing perspective. Information Systems Frontiers, v. 14, n. 2, p. 179-193, 2012. Available at: <http://www.springerlink.com/content/b7624v2u577214u1>. Acessed on: July 25, 2012. Archer, N. P.; Ghasemzadeh, F. An integrated framework for project portfolio selection. International Journal of Project Management, v. 17, n. 4, p. 207–216, 1999. Available at: <http://www.mmt3000.dk/Upload/N1%20An_integrated_framework_for_project_portf olio_selection.pdf>. Accessed on: March 11, 2012. Assis, C. B. Governança e gestão da tecnologia da informação: diferenças na aplicação em empresas brasileiras - dissertação de mestrado. Escola Politécnica da Universidade de São Paulo (Poli/USP). 2011.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

318 Heindrickson, G., Santos Jr., C. D.

Baron, R. M.; & Kenny, D. A. The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, v. 51, n. 6, p. 1173-1182, 1986. Available at: http://www.public.asu.edu/~davidpm/classes/psy536/Baron.pdf>. Accessed on: Oct. 5, 2012. BRASIL. Tribunal de Contas da União. Acórdão nº 1.603/2008, Plenário, 2008. Available at: <http://www.tcu.gov.br/Consultas/Juris/Docs/judoc/Acord/20080814/008380-2007-1-GP.doc>. Accessed on: May 3, 2012. BRASIL. Ministério do Planejamento, Orçamento e Gestão. Secretaria de Logística e Tecnologia da Informação. Instrução Normativa nº 04, de 2010, 2010. Available at: <http://www.governoeletronico.gov.br/sisp-conteudo/Legislacao>. Accessed on: Oct. 18, 2012. BRASIL. Tribunal de Contas da União. Sumários Executivos - Levantamento de Governança de TI 2010, 2010b. Available at: <http://portal2.tcu.gov.br/portal/page/portal/TCU/comunidades/tecnologia_informacao/ sumarios/Levantamento%20de%20Governan%C3%A7a%20de%20TI%202010.pdf>. Accessed on: May 20, 2012. BRASIL. Tribunal de Contas da União. Resolução TCU nº 247/2011, 2011. Available at: <http://www.tcu.gov.br/Consultas/Juris/Docs/judoc%5CResol%5C20120404%5CRES 2011-247.rtf>. Accessed on: Oct 12, 2012. BRASIL. Ministério do Planejamento, Orçamento e Gestão. Secretaria de Logística e Tecnologia da Informação. Guia para Criação e Funcionamento do Comitê de TI, 2011b. Available at: <http://www.governoeletronico.gov.br/biblioteca/arquivos/guiapara-a-criacao-e-funcionamento-do-comite-de-ti>. Accessed on: Oct. 10, 2012. BRASIL. Tribunal de Contas da União. Portaria TCU nº 156/2012, 2012. Available at: <http://www.tcu.gov.br/Consultas/Juris/Docs/judoc/PORTN/20120726/PRT2012156.doc>. Accessed on: Oct. 18, 2012. Castro, H. G. D.; Carvalho, M. M. D. Gerenciamento do portfolio de projetos: um estudo exploratório. Gestão & Produção, São Carlos, v. 17, n. 2, p. 283-296, 2010. Available at: <http://www.scielo.br/pdf/gp/v17n2/a06v17n2.pdf>. Accessed on: Oct. 5, 2012. Decancq, K.; Lugo, M. A. Weights in multidimensional indices of well-being: an overview. Leuven (Be): Center for Economic Studies, 2010. Available at: <http://www.econ.kuleuven.be/eng/ew/discussionpapers/Dps10/Dps1006.pdf>. Accessed on: Oct. 12, 2012. Dolci, P. C.; Maçada, A. C. G. The dimensions of IT portfolio management (ITPM): an analysis involving IT managers in Brazilian companies. JISTEM Journal of Information Systems and Technology Management, São Paulo, v. 8, n. 2, 2011. Available at: <http://www.jistem.fea.usp.br/index.php/jistem/article/view/10.4301%252FS180717752011000200005>. Accessed on: Oct 5, 2012.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

319

GAO. INFORMATION TECHNOLOGY INVESTMENT MANAGEMENT - A Framework for Assessing and Improving Process Maturity. Washington, D.C. (USA): United States General Accounting Office, 2004. Available at: <www.gao.gov/cgi-bin/getrpt?GAO-04-394G>. Accessed on: Aug. 25, 2012. Hayes, A. F. Beyond Baron and Kenny: Statistical Mediation Analysis in the New Millennium. Communication Monographs, v. 76, n. 4, p. 408‐420, 2009. Available at: <http://dionysus.psych.wisc.edu/lit/Articles/HayesA2009a.pdf>. Accessed on: Oct. 5, 2012. Hayes, A. F. Mediate. My Macros and Code for SPSS and SAS, 2012a. Available at: <http://www.afhayes.com/public/mediate.pdf>. Accessed on: Oct 5, 2012. Hayes, A. F. Process: A Versatile Computational Tool for Observed Variable Mediation, Moderation, and Conditional Process Modeling [White paper]. My Macros and Code for SPSS and SAS, 2012b. Available at: <http://www.afhayes.com/public/process2012.pdf>. Accessed on: Oct 5, 2012. ISACA. Cobit 5 - A Business Framework for the Governance and Management of Enterprise IT. Rolling Meadows, IL (USA): ISACA, 2012. Available at: <http://www.isaca.org/COBIT/Pages/default.aspx>. Accessed on: Nov. 11, 2012. ISACA. Cobit 5 - Enabling Processes. Rolling Meadows, IL (USA): ISACA, 2012b. Available at: <http://www.isaca.org/COBIT/Pages/Product-Family.aspx>. Accessed on: Nov. 5, 2012. ITGI. Board Briefing on IT Governance, 2nd Edition. Rolling Meadows, IL (USA): IT Governance Institute, 2003. ISBN 1-893209-64-4. Available at: <http://www.isaca.org/KnowledgeCenter/Research/ResearchDeliverables/Pages/Board-Briefing-on-IT-Governance-2ndEdition.aspx>. Accessed on: June 23, 2012. ITGI. COBIT - Control Objectives for Information and related Technology. 4.1. ed. Rolling Meadows, IL (USA): Information Technology Governance Institute, 2007. Available at: <http://www.isaca.org/Knowledge-Center/cobit/Pages/Downloads.aspx>. Accessed on: Sept. 7, 2012. ITGI. Enterprise Value Governance of IT Investments - The Val IT Framework 2.0 Extract. Rolling Meadows, IL (USA): IT Governance Institute, 2008. Available at: <http://www.isaca.org/Knowledge-Center/Val-IT-IT-Value-Delivery-/Documents/ValIT-Framework-2.0-Extract-Jul-2008.pdf>. Accessed on: July 10, 2012. Lunardi, G. L.; Becker, J. L.; Maçada, A. C. G. Um estudo empírico do impacto da governança de TI no desempenho organizacional. Produção, v. 22, n. 3, p. 612-624, maio/ago 2012. Available at: <http://www.scielo.br/pdf/prod/v22n3/aop_t6_0007_0374.pdf>. Accessed on: Oct. 20, 2012. Machado, C. P. Governança da tecnologia de informação e a efetividade dos sistemas de informação. Tese de doutorado. Programa de Pós-Graduação em Engenharia de Produção da Universidade Federal do Rio Grande do Sul. 2007.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

320 Heindrickson, G., Santos Jr., C. D.

Mackinnon, D.P. Introduction to Statistical Mediation Analysis: Multivariate Applications Series. New York (USA): Lawrence Erlbaum Associates, 2008. Markowitz, H. Portfolio Selection. Journal of Finance, v. 7, n. 1, p. 77-91, mar. 1952. Available at: <http://www.jstor.org/stable/2975974>. Accessed on: May 15, 2012. Mcfarlan, F. W. Portfolio approach to information systems. Harvard Business Review, v. 59, n. 5, p. 142-150, set-out 1981. Available at: <http://hbr.org/1981/09/portfolio-approach-to-information-systems/ar/1>. Accessed on: Aug. 9, 2012. Mello, Gilmar R. Governança corporativa no setor público federal brasileiro. Dissertação (mestrado em ciências contábeis) — Universidade de São Paulo, São Paulo, 2006. Mendonca, Cláudio Márcio Campos de et al. Governança de tecnologia da informação: um estudo do processo decisório em organizações públicas e privadas. Rev. Adm. Pública, Rio de Janeiro, v. 47, n. 2, 2013. Moraes, R. O.; Laurindo, F. J. B. Um Estudo de Caso de Gestão de Portfolio de Projetos de Tecnologia Da Informação. G&P Gestão & Produção, v. 10, n. 3, p. 311328, dez. 2003. Available at: <http://www.scielo.br/pdf/gp/v10n3/19165.pdf>. Accessed on: April 21, 2012. Peppard, J.; Ward, J.; Daniel, E. Managing the realization of business benefits from IT investments. MIS Quarterly Executive, v. 6, n. 1, p. 1-11, 2007. Available at: <http://misqe.org/ojs2/index.php/misqe/issue/view/21>. Accessed on: Oct 15, 2012. PMI. The Standard for Portfolio Management. 2nd. ed. Newtown Square, Pennsylvania (USA): Project Management Institute, 2008. Ramos, K. H. C. Governança e Gestão de Tecnologia da Informação: decompondo a organização em componentes com base em arquitetura organizacional orientada a serviços. Dissertação de Mestrado em Engenharia Elétrica, PPGENE.DM-410/09, Departamento de Engenharia Elétrica, Universidade de Brasília, Brasília, DF, xii, 203 p. 2009. Tavakol, M.; Dennick, R. Making Sense of Cronbach’s alpha. International Journal of Medical Education, v. 2, p. 53-55, 2011. Available at: <http://www.ijme.net/archive/2/cronbachs-alpha.pdf>. Accessed on: Sept. 21, 2012. Ward, J. M. A Portfolio Approach to Evaluating Information Systems Investments and Setting Priorities. Journal of Information Technology, v. 5, p. 222-231, 1990. Available at: <http://dx.doi.org/10.1057/jit.1990.46>. Accessed on: Jan. 18, 2012. Weill, P.; Ross, J. W. Governança de TI: Tecnologia da Informação. Revisão Técnica: Tereza Cristina M. B. Carvalho. São Paulo: M. Books do Brasil Editora Ltda., 2006. Weill, P.; Woerner, S. L.; Mcdonald, M. Managing the IT Portfolio (Updated Circa 2009): Infrastructure Dwindling in the Downturn. MIT Center for Information Systems Research - Research Briefing, Cambridge-MA (USA), v. IX, n. 8, Aug. 2009.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

321

Weill, P.; Woerner, S. L.; Rubin, H. A. Managing the IT Portfolio (Updated Circa 2008):It's All About What's New. MIT Center for Information Systems Research Research Briefing, Cambridge-MA (USA), v. VIII, n. 2B, July 2008.

APPENDIX A – Formulated questions and research variables 1. Questions about (PROCPORTF):

the

investment

portfolio

management

process

1.1 The selection and prioritization of proposals for new IT solutions are based on clear and transparent criteria, previously known to all parties involved. 1.2 The selection and prioritization of proposals for new IT solutions take into account the strategic, financial, pro-improved performance and informational benefits that can be generated by each IT solution for the business. 1.3 The selection and prioritization of proposals for new IT solutions take into account the costs related to financial disbursements and the costs associated to their own personnel allocated to develop or contract, support and operate the IT solution, estimated both for the project phase, as well as for the operation phase. 1.4 The analyses associated to the benefits, costs and risks of new IT solutions are recorded/documented, in order to provide input for the monitoring of these elements during the project phase, as well as during the solution’s operation phase. 1.5 The responsibilities of the IT area, of the demanding (interested) unit and of the IT steering committee or equivalent body regarding the selection/prioritization stage, as well as the operation and support stage of IT solutions, are well defined. 2. Questions about the IT Steering Committee or equivalent structure (COMITE): 2.1 The organization counts on one (or more) IT steering committee or equivalent structure with representatives from business units that engage in the institution’s governance and IT management. 2.2 The representatives of business units in the IT steering committee are heads of organizational units and not advisors or middle-management staff. 2.3 The IT steering committee decides (or is always consulted) on the selection and prioritization of IT projects/initiatives aimed at promoting new IT solutions. 2.4 The IT steering committee follows up/monitors the fulfillment of benefits, costs and risks of the most important IT solutions, during the project phase and during the operation and support phase. 3. Questions about IT solution managers or equivalent mechanism (GESTSOL): 3.1 There is a clearly defined management unit for each one of the IT solutions considered relevant or essential for the institution.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

322 Heindrickson, G., Santos Jr., C. D.

3.2 The IT solution management unit preferably is the owner of the business process affected by the solution or is the demanding unit or is the unit most interested in the solution. 3.3 The management unit defines the functional, non-functional and security requirements of the IT area, as well as the business rules of the IT solution. 3.4 The managing unit, together with the IT area, defines the service levels for the IT solution and monitors their fulfillment for as long as the solution remains in operation. 3.5 The managing unit monitors the fulfillment of benefits, costs and risks during the operation phase of the IT solution. 4. Questions about satisfaction towards IT governance and management (EFETGOVTI): 4.1 The managers of internal units, IT area clients, and other stakeholders are satisfied with the selection and prioritization method for new IT solutions. 4.2 Internal clients and other stakeholders are satisfied with the number of demands per new IT system/solution that have still NOT been met by the IT area (pending requests or backlog). 4.3 Internal clients and other stakeholders are satisfied with the swiftness (deadlines) agreed upon and effectively fulfilled by the IT area for the delivery of new IT solutions. 4.4 Internal clients and other stakeholders are satisfied with the costs agreed upon and effectively fulfilled by the IT area for the delivery of new IT solutions. 4.5 Internal clients and other stakeholders are satisfied with the quality and the level of service presented by the IT solutions currently in operation. 4.6 Internal clients and other stakeholders are satisfied with the fulfillment of strategic, financial, performance improvement and information benefits provided by IT solutions.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

323

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

APPENDIX B – Other properties of the sample Average

Standard “p” value deviation (bilateral)

Respondents from other inst. 117

9.2

1.1

TCU respondents

9.1

1.0

Managers

9.2

1.0

Non-managers

9.1

1.1

With IT Gov. course

131

9.4

0.8

Without IT Gov. course

8.8

1.5

IT personnel

151

9.1

1.1

IT area clients

9.4

0.9

IT auditors/consultants

9.3

0.9

Non IT auditors/consultants

105

9.1

1.2

Group

Qtty

0.553 0.551 0.036 0.088 0.070

Obs.: Because it refers to a non-normal, right-tailed, negative-skewed distribution, the Wilcoxon-Mann-Whitney hypothesis test was used in substitution of Student’s t-test. The hypotheses tested are: H0: The difference in the position between the two samples/groups is equal to 0. H1: The difference in the position between the two samples/groups is different from 0 Table B1 – Perception of groups regarding the importance of the IT Portfolio Process

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

324 Heindrickson, G., Santos Jr., C. D.

Average

Standard “p” value deviation (bilateral)

Respondents from other inst. 117

9.0

1.3

TCU respondents

8.8

1.1

Managers

9.0

1.1

Non-managers

8.9

1.3

With course on IT Gov.

131

9.1

1.0

Without course on IT Gov.

8.6

1.5

IT area personnel

151

9.0

1.2

IT area client personnel

8.9

1.1

IT auditors/consultants

9.2

0.9

8.7

1.3

Group

Qtty

Non-IT auditors/consultants 105

0.036 0.520 0.050 0.387 0.004

Obs.: Because it refers to a non-normal, right-tailed, negative-skewed distribution, the Wilcoxon-Mann-Whitney hypothesis test was used in substitution of Student’s t-test. The hypotheses tested are: H0: The difference in the position between the two samples/groups is equal to 0. H1: The difference in the position between the two samples/groups is different from 0 Table B2 – Perception of groups regarding the importance of the IT Steering Committee

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

325

Information Technology Governance In Public Organizations: How Perceived Effectiveness Relates To Three Classical Mechanisms

Average

“p” Standard vallue deviation (bilateral)

Respondents from other inst. 117

9.0

1.2

TCU respondents

8.9

1.0

Managers

9.1

0.9

Non-managers

8.9

1.2

With course on IT Gov.

131

9.1

0.8

Without course on IT Gov.

8.7

1.4

IT area personnel

151

9.0

1.1

IT area client personnel

9.0

0.9

IT auditors/consultants

9.3

0.8

8.8

1.2

Group

Qtty

Non-IT auditors/consultants 105

0.218 0.428 0.038 0.496 0.006

Obs.: Because it refers to a non-normal, right-tailed, negative-skewed distribution, the Wilcoxon-Mann-Whitney hypothesis test was used in substitution of Student’s t-test. The hypotheses tested are: H0: The difference in the position between the two samples/groups is equal to 0. H1: The difference in the position between the two samples/groups is different from 0 Table B3 - Perception of groups regarding the importance of the IT Solution Manager

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

326 Heindrickson, G., Santos Jr., C. D.

Group

Cause Cause Cause Cause Cause Cause 1 2 3 4 5 6

TCU respondents 56% 24% 69% 8% 53% 40% Respondents from other 80% 46% 83% 43% 52% 57% inst. Managers 76% 39% 76% 35% 52% 49% Non-managers 67% 38% 80% 26% 53% 52% With course on IT Gov. 72% 39% 81% 33% 52% 51% Without course on IT Gov. 71% 37% 71% 25% 54% 50% IT area personnel 79% 45% 79% 34% 47% 55% IT area client personnel 42% 9% 76% 18% 76% 33% IT auditors/consultants 68% 39% 78% 41% 48% 50% Non-IT 75% 38% 78% 22% 57% 52% auditors/consultants Cause 1: Insufficient IT personnel Cause 2: Lack of technical training by IT personnel Cause 3: Lack of or shortcomings in IT governance and management processes Cause 4: Insufficient financial/budgetary resources Cause 5: Alternatives regarding the provision of IT solutions are not sufficiently exploited (ex: in-house vs. outsourced development) Cause 6: Shortcomings in management skills training for IT personnel Obs: Respondents were allowed to select multiple causes or none of these causes; therefore, the variation possible for each of the causes is 0 to 100% for each stakeholder group. Table B4 â&#x20AC;&#x201C; Perception of groups regarding the main causes of non-effectiveness of IT Governance

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 297-326

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 327-344 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200006

THE EFFECT OF THE INTERNET ON THE PATIENT-DOCTOR RELATIONSHIP IN A HOSPITAL IN THE CITY OF SÃO PAULO Jayr Figueiredo de Oliveira EAESP-FGV – Fundação Getúlio Vargas, São Paulo, SP, Brasil _____________________________________________________________________ ABSTRACT Internet use by patients as a source of information on health and disease is expanding rapidly with obvious effects on the doctor-patient relationship. A self-administered questionnaire was used, about physicians´ perception on the use of the Internet by patients and about whether that use interferes with the doctor–patient relationship. 85.3% of the physicians reported that their patients accessed the Internet and that 92% used that information in a following visit. Overall, 56.9% of the physicians thought that the Internet helped the doctor–patient relationship, 27.6% thought it interfered with the relationship, and 15.5% believed that the Internet had a negative impact on it. Keywords: Internet, human relationship, health services, doctor–patient relationship, health information

1. INTRODUCTION Widespread use of the Internet as a source of health information has had an effect on health knowledge, attitudes and practices of the general population as well as on the doctor-patient relationship. With hundreds of thousands of health-related websites and millions of health-related computer searches conducted each day (Eysenbach, 2003), the possibilities for rapid changes in health behaviors are enormous. The majority of health-related Internet searches by patients are for specific medical conditions. They are carried out before the clinical encounter, to seek information to manage their own health care independently, to decide whether they need professional help, or to overcome reticence in discussing personal issues. Searches are also carried out after the encounter for reassurance or because of dissatisfaction during the encounter (McMullan, 2006) The fact that medicine has taken many turns in recent decades is not a novelty; the new reality is that patients have also changed. Medicine in former times, more of a craft than a science then, has today become technological. Patients, who in past times might have relied blindly on doctors, now attend an appointment having already provided themselves with information and vocabulary previously restricted to medical professionals – although, in some cases, they are only there to get a prescription from the doctor. Who is responsible for this transformation? We suggest it is the growing _____________________________________________________________________________________________ Manuscript first received/Recebido em: 21/02/2013 Manuscript accepted/Aprovado em: 26/06/2014 Address for correspondence / Endereço para correspondência Jayr Figueiredo de Oliveira, Bacharel e Mestre em Administração (PUC-SP), Doutor em Educação (PUC-SP), PósDoutor em Administração (EAESP-FVG). Pesquisador da EAESP-FGV – Fundação Getúlio Vargas, São Paulo, SP, Brasil, Av. 9 de Julho, 2029, Bela Vista, São Paulo – SP E-mail: profjayrfigueiredo@gvmail.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

328 Oliveira, J. F.

influence of information and communication technology (ICT), and in particular the Internet, that is at the root of the change. Access to websites with vast – but not always reliable – amounts of information on all sorts of diseases, as well as discussion forums, virtual communities, e-mail exchanges, and whatever else global search engines can offer, have contributed to revolutionizing the doctor–patient relationship. But are medical professionals prepared for this change? How to behave when patients believe they know everything about their illnesses? Should physicians participate in forum discussions and virtual communities or not? Should they send e-mails to patients or not? What attitude should a physician adopt toward a patient who fits this new profile? Given the reality of this new patient behaviour profile, physicians and the most modern hospitals in the developed world have made many improvements in their use of ICT, with the latest hospital management processes fully computerized and technology applied in all phases of patient care (Valle, 2010). The implementation of electronic patient records is growing so dramatically that in a short period of time – about two to five years – it will allow all patients, in both the public and private hospitals of the city of São Paulo, to wear barcode wristbands, ensuring that physicians and nurses have immediate access to diagnosis and treatment information. In addition, hospitals are working towards specialized diagnosis and treatments for high complexity diseases and investing heavily in new technologies. In the future, always bearing in mind the role of personalized assistance and the supply of specific services, hospitals will be better prepared to serve patients in a holistic manner. The hospitals of the future should be better able to allocate human and technological resources to biological, psychological, and social care. According to Hummel (2006), the intensive use of ICT will change the role of clinical staff (doctors, nurses, physiotherapists, and others) who will no longer perform mechanical tasks to take care of patients; instead, they will mainly offer emotional and psychological support. On the other hand, Hay, Cadigan, and Khanna (2009) claim that there remains an expectation of future hospitals having an increasing combination of technology and alternative therapies, and thus a path to prioritizing humanizing roles. Environmental and psychosocial activities and a fusion of Eastern with Western medicine will also contribute to a healing process towards quality of life improvement. According to a survey by the World Health Organization (WHO), Brazil is the second largest producer of medical technology in emerging countries, second only to China. Brazilian industry sales in 2009 topped US$2.6 billion in this sector, ahead of Mexico, India, and Turkey, which respectively held third, fourth, and fifth places in this ranking. The document released by the WHO in September, 2010, points to a greater participation of the emerging markets in medical technology. However, the medical equipment industry remains concentrated in developed countries. The United States tops the list with US$91.3 billion sales in 2009, an amount equivalent to 40.7% of the market. Next, come Japan and Germany which account respectively for 10.1% and 8.1% of total sales in 2009 (WHO, 2010). Between March 2003 and March 2010, Brazil invested more than US$6 billion in infrastructure, research, and technology in the healthcare industry. These resources came from the Federal Government, the Brazilian Development Bank (BNDES), and research support agencies. Despite this progress, the country still has an approximate US$9 billion trade deficit in the healthcare sector, according to 2009 data. Even so, the early investment results show a scenario allowing an optimistic view of the sector JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of São Paulo 329

(WHO, 2010). According to the WHO, the medical technology market grows by about 6% per year. Total sales in 2008 amounted to US$210 billion, double the amount recorded in 2001. The sector employs about one million people (WHO, 2010). However, medical advances have not been recognizably associated with an increase in physicians’ satisfaction with their career (Smith, 2001). As reported by Seckin (2010), patients are dissatisfied with their doctor–patient relationship. These stated factors, coupled with concerns on medical care expenditure increase and the emergence of conflicting issues between users and healthcare service providers have generated renewed interest in studies and reflections on the doctor–patient relationship. In this context, particularly the changing dynamics of performance in the doctor–patient relationship, this article aims to identify and analyze the impact of Internet use, particularly on the human side of that relationship. The discussion is relevant to the presence of new technologies influencing the quality of care and debates about the role of the physician in the current working environment. 2. LITERATURE REVIEW The widespread use of the Internet as a source of health information has caused a knowledge outcome towards attitudes and practices of people in general, as well as in the doctor–patient relationship. Thus, thousands of websites and millions of daily Internet health-related information searches (Kim & Kim, 2009) enable an enormous and rapid change in health behaviour. Most Internet searches performed by patients before their clinic appointment are directed to specific clinical conditions. Information is sought with the objective of dealing independently with their own medical care, thereby being able to decide whether there is a need for professional help or not, and in some cases, overcoming reluctance to disclose personal matters. Searches may continue even after the appointment because of a lack of satisfaction with the most recent medical consultation or even to confirm the validity of what they have been told by their doctor.” (Grossman et al., 2009). Along with the benefits of easily accessible information, there are concerns about the quality of the obtained content and its effects on the doctor–patient relationship (Kim & Kim, 2009). These concerns include unreliable information, which consequently may result in misguided self-treatment and wrong diagnosis; language barriers; commercial biases; longer clinic visits, resulting from disagreements on the patient’s information presented to the health care professional; unnecessary tests and treatments resulting from the patient’s Internet search (Walker & Carayon, 2009). Thus, the physician’s insight, experience, and encouragement are necessary to help the patient to accurately interpret and apply this information. Major approaches to the doctor–patient relationship The role of medicine in the second half of this century has been re-analyzed within several perspectives. In the late 1950s, the German physician and philosopher Karl Jaspers (1958/1991) developed a series of reflections on medical professionals in the age of technology with a strong criticism of psychoanalysis. He emphasized the need for medicine to recover the subjective elements of communication between doctor and patient that were improperly assumed by psychoanalysis and left aside by medicine,

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

330 Oliveira, J. F.

thus pursuing a path exclusively based on technical instrumentation and data objectivity. Awareness of the need to develop a communicative interaction between doctor and patient started to expand in the 1960s through medical psychology studies (Coelho Filho, 2007) of doctors’ psychoanalytic analysis (Montana et al., 2012), as well as the Balint groups’ experience while introducing the psychological dimension of the doctorpatient relationship, and the psychotherapeutic training needed for physicians (Seckin, 2010). Among several other communication theories, must be noted the School of Palo Alto and some key members of the renowned “Invisible College”: Gregory Bateson, Watzlawick, and Jackson (Watzlawick et al., 1972). During the 1960s and through the 1970s, Talcott Parson’s work dealing with the doctor–patient relationship and intentional consensus, which pioneered the field of health sociology and is now available as “informed consent,” originated from exigencies of consumers’ rights protection. A very current need to reduce the harmful effects from doctors´ improper behaviour towards patients has resulted in an increase of complaints against them in various countries, parallel with the increase in health expenses. Thus, in the search for expenditure reduction, several studies have dealt with the quality of health services and guidelines for reorganizing the healthcare model, including users’ points of view on service provision as presented by the healthcare system (Goodman, 2010). Most of these studies are based on the work of Donabedian, who in the early 1980s published several volumes and articles on this issue (Donabedian, 1990). A survey by Boltanski (1979) dealt with doctor–patient communication in various regions of France. Boltanski discusses the differences between scientificmedical knowledge and family-medical knowledge and relates these differences to the doctor–patient relationship. This relation between professionals and the diseased is regarded as a source of anxiety, mainly by the lower classes, since they usually have no objective criterion of evaluation available to them, thus highlighting standard difficulties with communication and chiefly referring to the doctor as “not being honest.” Boltanski’s text, was used as a basis for many studies conducted in Brazil. Another perspective presented by Russ et al. (2011) analyzes the doctor–patient relationship from an anthropological point of view, not only the cultural component of the disease, but also the experience and point of view of patients and relatives, interpretations, popular practices, and their resulting influence on prevention, diagnosis, and treatment. These authors’ research has had a large influence on the course on doctor–patient communication offered at Harvard Medical School (Branch et al., 1991), as well as other training programs (Seppilli & Caprara, 1997), resulting in a key component in undergraduate training and postgraduate studies in the medical field (Usherwood, 1993). Contemporaneously, the doctor–patient relationship has been targeted as a key aspect of improving healthcare quality and divided into several components such as personalized care, humanization of care, and right to information (Goodman, 2010), dealt with through topics such as users’ degree of satisfaction with healthcare services (Taha et al., 2009), counseling (Godfrey & Johnson, 2008), physician–patient communication (Kivits, 2009), patient suffering, biomedicine’s purpose (Giveon et al., 2009), and informed consent (Goodman, 2010). Among Brazilian authors, such as Salles (2010), Sucupira (1982), Coelho Filho (2007), it is possible to observe that arguments related to this issue are discussed again; JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of São Paulo 331

however, these have been in the form of essays presenting opinions or declaring theoretical inspirations. Two works stand out as being based on a systematic analysis to a large extent. Sucupira’s (1982) study would be the first of this kind to be developed. At the time the study was conducted, the Brazilian population was served primarily by three systems operating in parallel: social security, medical groups, and state and municipal health centers. Sucupira’s analysis, focusing on childcare assistance, indicates consumers’ choice patterns, resulting from the evaluation of a problem which triggered a search for service, in the context of the care system characteristics at the time. This research, in the view of the consumer, took into consideration physical conditions, accessibility, efficiency, professional availability, and medicines. For example, it was considered that the best childcare could only be found at the National Institute of Health and Social Security (INAMPS), and not at the State Medical Center. The main reason was that at the INAMPS it was always possible to visit the same doctor. This was evidence that these three big providers had different models for health care, which were recognized by the public, thus providing an option for them to choose the most suitable model to their needs in their search for care services, including relational dimension aspects. In his new environmental analysis, Goodman’s (2010) study is particularly relevant. He interviewed doctors with extensive clinical practice to examine the way they included technology in their professional activities. This analysis, from a pronounced historical-structure perspective, demonstrates that doctors, unlike other professionals, have not lost the property of “knowing” and “applying” when faced with the capitalist model of consolidation. However, parallel to this social transformation, we can observe the valuing of science and the intellectualization of knowledge. Medicine would have gone through the universalization of its actions, having patients as the object of their cognition, and in such circumstances, social differences would be left aside, giving priority to the object of scientific wisdom. In such conditions, the medical process is configured as a repetitive act of knowledge enabled by science, having thus entered the world of serial production, a factor which marks the industrial technology society (Goodman, 2010). Confidence in the doctor–patient relationship The focus of power in health care is shifting: instead of the doctor acting as sole manager of patient care, a consumerist model has emerged in which patients and their doctors are partners in managing their care. A number of studies focus on whether the Internet can actually empower patients and enrich the patient-doctor relationship (Shoor and Lorig, 2002; Sinclair, 2007). Trust has been described as one of the scarcest medical products (Attfield et al., 2006). The era of “the doctor knows best” lasted for most of the twentieth century because of the lack of information. However, with the arrival of the information age, patients came under the influence of the digital revolution. The immediate reaction has been that confidence in the doctor is replaced by skepticism and discouragement. “Blind trust” is replaced by “informed trust.” In fact, the first medical contact, which traditionally was with the family doctor, is being slowly replaced in many cases by the Internet. Patients search on medical websites and then consult their doctors armed with that information. A study of 500 “health seekers” revealed that 55% of them obtained information online before visiting the doctor and that 32% sought information about a particular doctor or hospital (Fox & Rainie, 2002). Among those who considered their JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

332 Oliveira, J. F.

search of websites a success, 38% reported that it led them to ask their doctor new questions or even opt for a second opinion (Fox & Rainie, 2002). â&#x20AC;&#x153;Virtual patientsâ&#x20AC;? across Europe also seem to be following this trend; a recent Internet study of a website in five languages found that among 6.699 European respondents, 73% indicated doctors as their favourite source of health information, although 45% also used the Internet, while 19% mentioned the Internet and not the doctor as a favourite source (Health and Age, 2011). According to the Research Center for Information and Communication Technology, among 63 million people in 2009 using Internet access in Brazil, 39% were people living in large urban centers looking for health information, representing a growth of 6% per annum among people interested in this type of query. Grosseman and Stoll (2008), however, warn that a large resistance from healthcare professionals has been noticed against the modification of the doctorâ&#x20AC;&#x201C;patient relationship dynamics in the age of information, owing to a bigger concern on the inconsistent and seldom reliable medical information on the Internet, the waste of the human factor, and the problematic perception of the informed patient. Contribution of ICT in the Medical Field In a complex universe formed by multiple and diversified organizations, ICTs have become competitively instrumental in driving and positioning virtually any organization, often transforming the reality and essence of the business itself. Given the enormous potential, organizations are leading the development and application of ICTs, either through the optimization of internal works or by inducing changes at the business level, thereby capitalizing on ICT developments in order for them to become more dynamic and better qualified to innovation in response to market changes (Hummel, 2006). Propelled at first through successive technological advances, in terms of equipment and software, then through an increasing recognition of its potential, ICTs have emerged from the gloomy condition of doing mainly automation tasks in organizations, in order to make an up-to-date assessment (Hummel, 2008) as a determining factor for competitive positioning. ICTs are the bedrock of contemporary organization. Currently, it is almost impossible to conceptualize an organization that does not use ICT; it would not be excessive to say that the effects of ICT have been (and certainly will continue to be) fully integrated in organizations (Valle, 2010), whether from the point of view of incorporating these technologies into the value chain of the enterprise or from the point of view of establishing a competitive advantage. As an important catalyst for change in the manner that work is performed, information technologies also play a key role in the health sector (Hummel, 2008). As a whole, the advent of the Internet has led this industry to exploit ICTs in order to improve patient care in hospitals, increase the commercial effectiveness of institutions, and optimize communication between the various players in the sector (MacGregor et al., 2010). As MacGregor et al. (2010) state, several studies identify the potential benefits of ICT use in the health sector. For instance, Akersson et al. (2007) refer to service quality improvement and information availability. El-Sayed and Westrup (2003) suggest that ICT use in hospital practices improves communication, causing business effectiveness growth and contributing to new enterprise initiatives.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of SĂŁo Paulo 333

Seckin (2010) suggests that ICT supports and allows complex interactions between, for example, doctors, nurses, patients, and sometimes facilities. Hummel (2008) suggests that when ICT is applied to medical practices, it produces a change in procedures which result in a more effective final product. Goodman (2010) reports an increase in the financial income. Ray and Mukherjee (2007) note the use of ICT in management and planning development. Other studies such as the ones by Lougheed (2004) and Ho et al. (2004) show that the current use of ICT transcends the context of clinical tools, incorporating a wider role for management practice tools in business. It becomes clear that, without an efficient and effective use of ICT, organizations cannot be efficient or competitive, and in most cases, they depend on these abilities to survive (Hummel, 2006). Given the importance of the Internet today in the context of organizational development, it is essential to characterize the current reality in terms of its adoption in hospitals, in the interest of improving evaluation and future planning. Technological Investment in the Health Sector An increasing worldwide emphasis in the health sector has been observed in recent years, mainly among Western countries and the United States, where a growth of per capita expenditures on health has been witnessed (OECD, 2008). This change is related to many aspects of the contemporary society, including increase in life expectancy, and demand from people for a better quality of life and better health services. At the same time, we are witnessing efficient approaches in terms of costs and new solutions based on ICT. In many Western countries, the health industry sector is large (Salles, 2010). Expenditures in the United States were US$2.2 trillion in 2007, accounting for 16% of GDP, with an expected growth rate of 6.2% per year through 2018, reaching US$4.4 trillion, corresponding to 20% of GDP in 2018 (Medicare.gov, 2007). Meanwhile, in Europe (EU-25) in 2006, total health expenditures rose to 7.76% of GDP; the EU-15 the average was 8.6% and in the new Member States 5.8 % (OECD, 2008). However, many factors are contributing to this increase in expenditure on health matters. The main factor is a combined effect of a predictable increase in the aging population along with a tendency for per capita expenditures on health to increase with age (Hummel, 2008). However, life expectancy increase and health care charges do not have a linear relationship as people not only require longer periods of treatment, but also frequency and intensity of health care increases. According to Salles (2010), the intention to computerize and mechanize medicine was commendable: it was thought that this would grant the doctor better conditions to be faster and more efficient, thus allowing an increase in time spent on the patient and on other work. It would also permit faster and more accurate laboratory tests â&#x20AC;&#x201C; especially those applying higher technology such as MRIs, CT scans, and genetic research, among other advances strictly connected to medical practice. However, an element that should have been helping medical practice instead began to control it; the medical activity of value became the technology-aided information search, while doctors started to become an obsolete article rather than a profit generator. Pressured by the healthcare administrations of hospitals, clinics, and even healthcare systems, doctors are nowadays required to see a predetermined number of patients per month, and if they do not reach this quota, they are considered incompetent JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

334 Oliveira, J. F.

and unproductive (Salles, 2010). The physician must work in a scheme that does not provide the necessary time for listening to, or developing a rapport with, the patient and then the patient’s trust – a necessary condition for a query to be considered satisfactory in any area of healthcare assistance. In the few predetermined minutes of a healthcare query, the physician hardly has time to look at the patient’s name and discover the patient’s needs through a conversation; instead, the physician has to proceed with standard questions that in turn merely standardize the patient. This process dismisses the clinical examination and arrives at a diagnosis primarily through laboratory tests (Salles, 2010). Moreover, science and technology have lately been providing healthcare professionals with a wide range of “tools.” Advances in pharmacology, material science, nanotechnology, ICT, and medical practice, contribute to new or improved healthcare mechanisms. Orientation towards innovation and competitiveness among enterprises has also pushed the development of many new medical devices and solutions (European Commission, 2007). The Brazilian market has grown in the area of health technology (public and private); the merging of Brazilian companies with foreign companies to attain the right to technological patents has alone generated billions of dollars. However, Brazilian laws do not allow foreign investment in healthcare (Valle, 2010; Hummel, 2008) The Doctor–Patient Relationship and Information Access in the Internet Age Scientific and technical information access generally, coupled with the increase in the educational level of the population, has created a class of patients who seek information on illness, symptoms, medications, and costs of hospitalization and treatment. There has also been an increase in information availability and access in the field of health care, substantially through the Internet (Pereira Neto & Guilam, 2008). The main feature that differentiates the Internet from other media is the plethora of information sources and the dynamic stance of the user. This information obtained through the Internet modifies or interferes with the doctor–patient relationship (Pereira Neto & Guilam, 2008). Doctors are no longer the sole holders of supreme knowledge; rather, they are now being questioned by increasingly well-informed patients (Kim & Kim, 2009). People can now use the Internet to look up information on their complaints or on diseases they already know they have, and they can do this before going to their medical consultation. Usually the main point of entry is the Google search engine (Siempos et al., 2008). From there, they dive into specialized websites and blogs (Russ et al., 2003; Tang & Ng, 2006). It is estimated that half of the patients coming to the doctor’s office now bring information from the Internet. The doctor then has to realize that the patient has this information, and then patients should realize that the doctor has the information and the training and ability to apply that information (Pereira Neto & Guilam, 2008). 3. RESEARCH METHODOLOGY A descriptive exploratory study was conducted with a cross-sectional quantitative approach. The investigated population was composed of doctors performing professional activities in a large hospital in the central part of the city of São Paulo, Brazil - carried out between the second half of 2011 and first half of 2012. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of São Paulo 335

The survey population included physicians practicing in various specialties (obstetrics and gynecology, pediatrics, cardiology, and pulmonology, among others) caring for patients in their own offices or directly on the premises of the hospital. They agreed to be participants in the study through a consent form. Data were collected for three months through a self-administered questionnaire developed by the author, composed of open and closed questions on physicians’ perceptions on Internet use by patients and on whether this interferes with the doctor– patient relationship. The interviewees were anonymous, and identification data were treated as confidential by separating questionnaires from the consent forms in two different sealed envelopes. Data were entered into the EpiData program and analyzed with the SPSS version 19. Qualitative data were presented in absolute and relative values and quantitative data as measures of central tendency and dispersion. In order to test associations between the variables of interest, a chi-square test was used for categorical variables, and the oneway ANOVA to compare means, tested at the statistical significance level of 95%. The open-ended questions were transcribed and grouped as quotations, in order to illustrate the medical opinion of professionals on the subject in question. 4. RESULTS The study included 232 physicians, at an average age of 43.6 (SD ± 11.3) , ranging between 30 and 60 years. A total of 75.9% were male. Regarding the medical practice period, the average was 18.6 ± 10.6 years, ranging between 5 and 35 years on the job. Table 1 shows the participants’ specialties. Besides their medical specialization, 29.3% among them had a master’s degree and 6.9% a doctorate´s degree. Table 1 - Specialty of Medical Professionals Participating in the Study Medical specialty

Pediatrics

17.2

Gynecology and obstetrics

14.7

Gastroenterology

12.0

Clinical medicine

6.0

Cardiology

5.2

Endocrinology

5.2

Other *

39.7

* Other medical specialties: nephrology, neurology, angiology and vascular surgery, ophthalmology, dermatology, pulmonology, geriatrics, hematology, infectious diseases, family medicine, oncology, orthopedics, proctology, psychiatry, acupuncture, rheumatology, urology. Among the participating physicians, 96.6% usually accessed the Internet and 99.1% had an e-mail account. Overall, 85.3% of respondents thought that patients accessed the Internet to search for information on their medical condition or for information on the symptoms that caused them to seek medical attention. But within their own clientele, they estimated that less than 50% of the patients they treated had this attitude, according to the medical professionals interviewed. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

336 Oliveira, J. F.

When questioned as to whether patients accessed the Internet themselves after a doctor’s appointment, 95.7% of the physicians believed their patients access the Internet after treatment. However, physicians pointed out that in their opinion around 92% of patients use information obtained from the Internet to talk to their doctors at the next appointment. On the socioeconomic profile of patients, doctors believed that only 2.6% of patients from the Unified Health System (SUS) used the Internet. The vast majority (65%) were private or health plan patients, leading to the conclusion of a more widespread use of Internet searches among individuals with higher purchasing power. For 46.6% of respondents, the opinion was that patients who attended the doctor’s office after reading a lot of information on the Internet usually made their appointments last longer. However, a considerable number of physicians (31.9%) did not perceive this difference of time. Furthermore, 42.2% of the doctors stated that they had patients arriving with information from the Internet, which was new to even the professionals themselves. The Internet was also used as a tool for updating their professional knowledge by 94% of the interviewed physicians and 58.6% of them stated they liked to suggest trustable sites to their patients. When patients came to the clinic with a large amount of Internet-obtained information regarding their illnesses, doctors had different emotional reactions. Of the total, 50% felt motivated or happy, 25% felt indifferent, 14% had confused feelings, and 11% felt uncomfortable. Comparing average age to period of practice, younger and less experienced physicians had a higher percentage of negative feelings towards patients getting medical information on the Internet. However, this difference was not statistically significant (p> 0.05). Also, differences between gender in feelings were not detected (p = 0.6), nor by medical field (p = 0.3) or postgraduate status (p = 0.9). In the doctors’ opinion, 56.9% believed the Internet helped the doctor–patient relationship, 27.6% thought it interfered with it and 15.5% believed that the Internet had a negative impact on it. Among the professional opinions of those who believed that the Internet helped the doctor–patient relationship, highlight those who believe that it can improve the communication between practitioners and patients, helping to treat the disease. Presented below are illustrations of opinions from selected interviews with the doctors. To preserve confidentiality, the physicians are identified only by number, from M1 to M13 - and elected only to exemplify some of the opinions collected: M1: “It helps them to understand a little more about their disease; it also becomes easier to explain the treatment and its consequences.” M2: “The higher the level of information and knowledge the patient has about their disease, the better is their acceptance of the treatment. Patient participation is extremely important for making decisions; it is also necessary to share guidelines, so that the desired effect may result from the treatment.” M3: “It helps, since it solves doubts that sometimes are not reachable in a conversation.” In addition, some professionals considered the Internet an optimizing tool for acceptance and understanding of diseases, with patients thus becoming more favourable to medical care: JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of São Paulo 337

M4: “In cases of chronic diseases, it is an important tool; experience shared among patients makes them aware of the dimensions of their own disease.” M5: “... can be used as an auxiliary tool during consultation.” M6: “It is far better to have a discussion with a patient, when they are aware of their illness, aiming for the best approach through mutual sharing of responsibilities and decision making.” Among those doctors who believed that Internet information use by patients undermined the doctor–patient relationship, there were concerns over somatization symptoms or psychological interference resulting from misinformation or poor understanding: M7: “... patients may sometimes access sites with poor quality information or even interpret this data improperly, thus creating anxiety before the exact diagnosis.” M8: “In cases where a patient believes most of what they read on the Internet, this can induce them to feel the symptoms that they have read about, even if they don´t in fact exist.” M9: “Internet can be harmful, when patients create an expectation that can sometimes be disappointing when faced with the knowledge of an actual disease stage or erroneous information obtained through unreliable websites.” M10: “The large amount of fallible information on the Internet causes a negative effect on the acceptance of a disease by the patient.” There are some professionals who feared they were being “tested” on information they are unaware of: M11: “In some cases, a distrustful patient misuses the information obtained by trying to test the physician’s knowledge.” A considerable number of physicians (27.6%) believed that Internet use by patients, as a complementary method, did not involve either damage or benefit, i.e., this newly adopted attitude of many patients did not interfere with the doctor–patient relationship: M12: “Knowledge must be multiplied, democratized. The patient has the right to seek information on his disease wherever they want. There is good quality information to be found, as well as bad, but that should not interfere with the doctor–patient relationship.” M13: “Patient knowledge improvement and communication is made easier; therefore, there is no reason to believe that it may interfere with the doctor–patient relationship.” Doctor’s age and length of time in practice had no significant statistical association with either positive or negative opinions of the Internet on the doctor– patient relationship (p = 0.4 and p = 0.5, respectively), similarly for post-graduate qualification (p = 0.3). However, when comparing specialties, grouped by clinical and surgical categories, clinicians had a more favourable opinion of the Internet than surgeons, and this difference was statistically significant (p = 0.03).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

338 Oliveira, J. F.

Table 2 - Quality of Health Information on the Internet as Perceived by Doctors Opinion of health information on the Internet

N (%)

Important to understand the disease and adherence to treatment

136 (58.6)

Unnecessary, since most information is incorrect

30 (12.9)

It makes no difference; the patient should follow medical guidelines

16 (6.9)

Depends on the source investigated

50 (21.6)

Table 2 summarizes doctors’ opinions on information available on the Internet. 5. DISCUSSION Among the 232 physicians surveyed in this study, 85.3% answered that it was common for patients to access Internet information about their medical condition or about whatever other matter made them seek medical attention. This result is very similar to that of other studies, such as a Norwegian survey that discovered the proportion of the population seeking Internet information related to health increased from 31% in 2001 to 58% in 2005 (Andreassen et al., 2006). Estimates from other studies show that half of patients come to clinics with Internet information (Pereira Neto & Guilam, 2008). For example, a US study estimated that 70 million Americans used the Internet to obtain health-related information (Seckin, 2010). A Spanish study stated that around 40% of their population surfed the Internet for health information (Sobes, 2009); however, in this study it was estimated, according to medical professionals, that less than 50% of their patients did so during their treatment. It is likely that this difference between surveys occurred because of methodological differences in the studies. Also, the present study is a private estimate given by medical personnel and does not necessarily reflect what happens in practice, since patients may in fact access the Internet but do not mention it to their doctors. Comparing average age and period of practice, this study indicates that younger and therefore less experienced professionals had a higher percentage of negative feelings towards patients’ habit of searching medical information on the Internet. This opinion might be associated with the uncertainty of newly graduated doctors, due to their limited experience in professional practice. These professionals may fear being tested or not being aware of some information, as expressed by M11: “In some cases, a distrustful patient misuses the information obtained by trying to test the physician’s knowledge.” Comparing specialties grouped as clinical or surgical, clinicians had a more favourable opinion about the Internet than surgeons, and this difference was statistically significant (p = 0.034). Perhaps, this is justified because clinicians generally maintain a longer period of contact with patients, especially when treating chronic diseases; they also understand better how health information works to assist treatment adherence and lifestyle improvement.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of São Paulo 339

This study showed how common it is for patients to access the Internet after a medical consultation. Observations pointed to about 92% of patients using information obtained through the Internet, to use for discussion at their next doctor consultation. In another study with 3.209 participants, 31% had reported information seeking related to health, and only 8% among these used this Internet data to share with their doctor (Hay, 2009). There is a possible explanation for this new patient profile, and this is the guarantee of anonymity on the Internet. One of the positive aspects of this new change is that the patient is free to search about intimate aspects, thanks to the anonymity of the network, thus providing greater adherence to proposed treatments, and as a result, patients become more involved in medical consultations (Melamud et al., 2007). For this same reason, in the present study over half the physicians surveyed (58.6%) believed that the information found on the Internet was important for the patient to acquire medical knowledge on the disease, achieving adherence to treatment, and that this is a reason for improvement in the development of the doctor–patient relationship. Given the informal and accessible language used on the Internet, patients could achieve better understanding and be enabled to more effectively discuss treatment options with their doctor, thus also sharing the responsibility of decision making (Madeira, 2006; Gauld & Willians, 2009). When patients attend the doctor’s office after reading a lot of information from the Internet, the appointment lasts longer, as opined by 46.6% of the physicians surveyed. This probably occurs because patients who are in possession of a greater amount of knowledge on their disease have more curiosity and extra questions regarding the approach to be adopted by their doctors. However, a considerable number of physicians (31.9%) do not perceive this difference in consultation time, as justified by one of the doctors interviewed (M1): “It helps them to understand a little more about their disease; it also becomes easier to explain the treatment and its consequences.” Moreover, this new patient profile requires doctors to be constantly updating their professional knowledge (Pereira Neto & Guilam, 2008). It seems that it is not only patients who are turning to the Internet, since 94% of physicians in the present study used it to update themselves, 96.6% had Internet access, and 99.1% used email. As a result, 58.6% of physicians were suggesting reliable sites to their patients for responsible use. This approach to technology by some physicians may result from the aim to offer accurate and easily interpreted information access. Perhaps, there is also some fear among professionals that they may not be aware of latest information and they must face the fact that their career practice requires continued improvement. In one study conducted in Mexico, some cases were reported where patients had modified their behaviour after presenting medical information obtained on the Internet to their doctors (Herrera, 2008). In the present study, 42.2% of the doctors said that they were unaware of some of the Internet information presented by patients. This could be explained by the fact that some of this information was incomplete, contradictory, inaccurate, or even fraudulent. A Brazilian study in São Paulo assessed the impact of Internet use on the doctor–patient relationship, observing the danger of people thinking that information obtained on the Internet can be used to treat themselves without even consulting a doctor, as this would be unwise (Hummel, 2008). It is important to evaluate how patients use this Internet information, bearing in mind that patients may encounter difficulties understanding medical language as well as doubts arising from the multiplicity of information. It is indeed hard for a patient to distinguish right from wrong or even traditional procedures from novelties (Herrera et JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

340 Oliveira, J. F.

al., 2008). The consideration remains, therefore, whether patients are able to interpret in a proper manner the information they obtain. Among these negative aspects is a remarkable naivety in believing uncritically all available information from the Internet. That is why, despite the importance associated to the liberal and democratizing facets of the Internet, it is necessary to possess the technical and cultural prerequisites to access it intelligently. For example, we have (through physicians’ perception) a socioeconomic profile of patients who use the Internet: they are high-income individuals, represented by private healthcare or health insurance. Most patients find health information on the Internet to be as reliable as that from other sources, including information provided by physicians (Pereira Neto & Guilam, 2008; Marsiglia, 2006). Anyone surfing the net is liable to encounter criminal (counterfeit or prohibited drugs being sold, promises of miracle cures) or at least unethical activities (informative contents with the sole intention of selling drugs or medical procedures). It is also possible to find nonsensical information that may generate severe damage to health. Thus emerges the quality control problem of Internet content that is related to health (MacGregor et al., 2010). Of the 232 doctors interviewed, 15.5% believed that the Internet disrupted the doctor–patient relationship, and 12.9% considered the Internet information available to be unnecessary since much of it was incorrect. Among the limitations of this study is a lack of scientific literature on the topic. The focus of this research has therefore rested on the point of view of physicians regarding the new attitude adopted by patients. It is therefore of consequent importance to emphasize that the research is based on estimates taken from the personal impressions of doctors, and it does not necessarily reflect what happens in practice. Further studies may be needed to evaluate more precisely this new patient profile and its repercussions on the doctor–patient relationship. 6. CONCLUSION The role of information in determining a diagnosis or treatment is of fundamental importance to both doctor and patient, though with different influence factors for each. For medical professionals, there is the important role of training and updating of knowledge in their field; while for patients, the search for information may be a messy process that can exert either a positive or negative influence on the established doctor–patient relationship. In this study, the opinion of medical professionals was directed to the question of how the Internet affected the doctor–patient relationship. In the doctors’ opinion, 56.9% believed the Internet helped the relationship, 27.6% thought it interfered with it, and 15.5% believed that the Internet had a negative impact on it. As a conclusion, we observe that the theoretical frame of this question as an object of study still requires a significant amount of further research and published findings that tie together and comprehensively articulate the theory and empirical analyses and evaluations of proposed interventions. Regarding the doctor–patient relationship, there may need to be some rethinking of it, which places the patient in a position which is as active as the physician´s, to the extent that the patient’s complaint

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of São Paulo 341

guides the clinical moment. This rethinking of the place of the patient is essentially one of the targets of any project on the humanization of medicine. This research, however, should be a priority for a context supported by practical learning through reflection, without denying or belittling the technological resources in the daily lives of the profession, but using them as a resource and not a purpose of health interventions. REFERENCES Akersson, K.; Saveman, B. I.; Nilssom, G. (2007). Healthcare consumers’ experiences of information communication technology: A summary of literature. International Journal of Medical Informatics, 76(9), 633–645. Andreassen, H. K.; Trondsen, M.; Kummervold, P. E.; Gammon, D.; HJORTDAHL, P. (2006). Patients who use e-mediated communication with their doctor: new constructions of trust in the patient–doctor relationship. Qual Health Res, 16(2), 238– 248. Attfield, S.; Adams, A.; Blandford, A. (2006). Patient information needs: pre and post consultation. Health Informatics Journal 12, 165–177. Boltanski, L. (1979). As Classes Sociais e o Corpo. Rio de Janeiro: Edições Graal. Branch, W. T.; Arky, R. A.; Woo, B.; Stoeckle, J. D.; Levy, D. B.; Taylor, W. C. (1991). Teaching medicine as a human experience: A patient–doctor relationship course for faculty and first-year medical students. Annals of Internal Medicine, 114, 482–489. Coelho Filho, J. M. (2007). Relação médico–paciente: a essência perdida. Interface – Comunicação, Saúde, Educação, 11(23), septiembre-diciembre, 631–633. Retrieved July 7 2010, from http://redalyc.uaemex.mx/src/inicio/ ArtPdfRed.jsp?iCve=180115440018. Donabedian, A. (2007). La Qualita dell’Assistenza Sanitaria. Roma: NI. El Sayed, H.; Westrup, C. (2003). Egypt and ICTs: How ICTs bring national initiatives, global organizations and local companies together. Information Technology & People 16(1), 76–92. European Commission. (2007). eHealth – Priorities and Strategies in European Countries. Luxembourg: Office for Official Publications of the European Communities. Eysenbach, G. (2003). The impact of the 1. internet on cancer outcomes. CA Cancer J Clin, 53(6): 356-371. Fox, S.; Rainie, L. (2002). Pew Internet and American Life Project. “Vital Decisions”, Summary of Findings, and Part Six: Impact, Washington, DC, May. Retrieved April, 17, 2012 from http://www.pewinternet.org/pdfs/PIP_Vital_Decisions_May2002.pdf. Gauld, R.; Willians, S. (2009). Use of the Internet for health information: a study of Australians and New Zealanders. Inform Health Soc Care, 34(3), 149–158. Giveon, S.; Yaphe, J.; Igal Hekselman, I.; Mahamid, S.; Hermoni, D. (2009, September). The e-Patient: a survey of Israeli primary care physicians’ responses to patients’ use of online information during the consultation. IMAJ 11.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

342 Oliveira, J. F.

Godfrey, M.; Johnson, O. (2008). Digital circles of support: meeting the information needs of older adults. Comput Hum Behav, 25, 633–642. Goodman, K. W. (2010). Ethics, information technology, and public health: new challenges for the clinician–patient relationship. In: Symposium – the effects of health information technology on the physician–patient relationship. Journal of Law, Medicine & Ethics. Spring. Grosseman, S.; Stoll, C. (2008). O ensino–aprendizagem da relação médico–paciente: estudo de caso com estudantes do último semestre do curso de medicina. Revista Brasileira de Educação Médica, 32(3), 301–308. Grossman, J. M.; Zayas-Cabán, T.; Kemper, N. (2009, March/April). Information gap: can health insurer personal health records meet patients’ and physicians’needs? Health Affairs 28(2), 377–389. Hummel, G. S. (2006). eHelth: o iluminismo digital chega a saúde. São Paulo: STS. _______, G. S. (2008). ePatient: A odisseia digital do paciente em busca da saúde. São Paulo: STS. Hay, M. C.; Cadigan, R. J.; Khanna, D. et. al. (2009). Prepared patients: Internet information seeking by new rheumatology patients. Arthritis Rheum, 59(4), 575–82. Health and Age, (2011). “Europeans require more health information: Survey results from Europe.” Retrieved February 8, 2012, from www.healthandage.com. Herrera, C.; Campero, L.; Caballero, M.; Kendall, T. (2008). Relationship between physicians and HIV patients: influence on adherence and quality of life. Rev Saude Publica, 42(2), 249–255. Ho, K.; Lauscher, H. N.; Best, A.; Walsh, G.; Jarvis-Selinger, S.; Fedeles, M.; Chockalingam, A. (2004). Dissecting technology-enabled knowledge transition: essential challenges, unprecedented opportunities. Clinical and Investigative Medicice, 27(2), 70–78. Jaspers, K. (1958/1991). Il Medico nell’Età della Tecnica. Milano: Raffaello Cortina Editore. [ed.orig.: Der Arzt im technischen Zeitalter, Klinische Wochenschrift, 36 (1958), 1037–1143.] Kim, J.; Kim, S. (2009). Physicians’ perception of the effects of Internet health information on the doctor–patient relationship. Inform Health Soc Care, 34(3), 136–48. Kivits, J. (2009). Everyday health and the Internet: A mediated health perspective on health information seeking. Sociol Health Illn. 31(3), 673–687. Lougheed, T. (2004). Wireless points the way in Africa. Appropriate Technology, 31(3), 50. Macgregor, R. C.; Hyland, P. N.; Harvie, C. (2010). Associations between driving forces to adopt ICT and benefits derived from that adoption in medical practices in Australia. In: Handbook of research on developments in e-health and telemedicine: technological and social perspectives. IGI Global. Madeira, W. (2006). Navegar é Preciso: avaliação de impactos do uso da Internet na relação médico–paciente. [Dissertação de Mestrado]. Faculdade de Saúde Pública da USP. São Paulo: 132p. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

The effect of the Internet on the Patient-Doctor Relationship in a Hospital in the City of São Paulo 343

Marsiglia, I. (2006). Impacto de la tecnologia médica sobre la historia clínica y la relación médico–paciente. Gac Méd Caracas, 114(3), 183–189. Mcmullan, M. (2006). Patients using the Internet to obtain health information: how this affects the patient-health professional relationship. Patient Educ Couns, 63(1-2), 24-28. Medicare.GOV. (2007). National Health Expenditure 2008–2018, Forecast summary. Retrieved April 8, 2012 from http://www.cms.hhs.gov/NationalHealthExpendData/downloads/proj2008.pdf. Melamud, A.; Puiggari, J.; Goldfarb, G.; Cairoli, H.; Otero, P. (2007). Información sobre salud em internet: su empleo por pediatras y padres de pacientes. Arch Argent Pediatria, 105(6), 513–516. Montana, S. B.; Montana, G. L.; Moshavi, D. (2012). A knowledge-based view of improving the physician–patient relationship . Academy of Health Care Management Journal, 8(1), 9–19. OECD, OECD Health Data 2008. Retrieved April 13, 2012, http://ec.europa.eu/health/ph_information/dissemination/echi/echi_25_en.pdf.

from

Pereira Neto, A. F.; Guilam M. C. R. (2008). The Internet, expert patients and medical practice: An analysis of the literature. Interface comun. saúde educ., 12(26), 579–588. RAY, S.; Mukherjee, A. (2007). Development of a framework towards successful implementation of E-governance initiatives in health sector in India. International Journal of Health Care, 20(6), 464–483. Russ, H.; Giveon, S. M.; Catarivas, M. G.; Yaphe, J. (2011, April). The effect of the internet on the patient–doctor relationship from the patient’s perspective: a survey from primary care. IMAJ, 13. Salles, A. A. (2010). Transformações na relação médico–paciente na era da informatização. Revista Bioética, 18(1), 49–60. Seckin, G. (2010). Patients as information managers: the Internet for successful selfhealth care & illness management. Open Longevity Science, 4, 36–42. Seppilli, T.; Caprara, A. (1997). Il Rapporto Medico–Paziente: Un Approccio Antropologico, Corso di Formazione Specifica in Medicina Generale. Perugia: Fondazione Angelo Celli, Istituto di Etnologia e Antropologia Culturale, Università degli Studi di Perugia, Ordine Provinciale dei Medici-Chirurghi e degli Odontoiatri, Sezione Provinciale di Perugia della Società Italiana di Medicina Generale (SIMG), (mimeo). Shoor, S.; Lorig, K. R. (2002, April). Self-Care and the Doctor-Patient Relationship, Medical Care. 40(4) Supplement:II-40-II-44. SIEMPOS I, I.; SPANOS, A.; ISSARIS, E. A.; RAFAILIDIS, P. I.; FALAGAS, M. E. (2008). Nonphysicians may reach correct diagnoses by using Google: a pilot study. Swiss Med Wkly, 138, 741–745. Sinclair, F. In Need of TLC? A Doctor-Patient Relationship Fit for the Future, Scottish Council Foundation, 2007, http://www.scottishcouncilfoundation.org/page.php?id=25 Smith, R. (2001). Why are doctors so unhappy? BMJ, 322, 1073–1074.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 327-344

www.jistem.fea.usp.br

344 Oliveira, J. F.

Sobes, J. J. M.; Santacreu, G. L.; Martínez, S. L.; Remón, C. A. (2009). Uso de internet por médicos de primaria y hospitales y percepción de como influye em su relación com los pacientes. Atención Primaria, 41(6), 308–314. Sucupira, A. C. S. L. (1982). Relações Médico–Paciente nas Instituições de Saúde Brasileiras. Dissertação de Mestrado, São Paulo: Faculdade de Medicina, Universidade de São Paulo. Taha, J.; Sharit, J.; Czaja, S. (2009). Use of and satisfaction with sources of health information among older internet users and nonusers. Gerontologist Review, 49, 663– 73. Tang, H.; NG, J. H. (2006). Googling for a diagnosis: use of Google as a diagnostic aid: internet-based study. Br Med J, 333(7579), 1143–1145. Usherwood, T. (1993). Subjective and behavioural evaluation of the teaching of patient interview skills. Medical Education, 27, 41–47. Valle, A. B. et. al., (2010). Sistemas de informações gerenciais em organizações de saúde. Rio de Janeiro: FGV. Walker, J. M.; Carayon, P. (2009, March/April). From tasks to processes: the case for changing health information technology to improve health care. Health Affairs 28(2), 467–477. Watzlawick, P.; Helmick, J. B.; Jackson, D. D. (1972). Une Logique de la Communication. Paris: Éditions du Seuil. WHO (World Health Organization) (2010). Doctor–Patient Interaction and Communication. Geneva: Division of Mental Health. WHO.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.327-344

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 345-360 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200007

STATE OF THE ART: INTEGRATING SERVICES FOR MEGA EVENTS Jorge R. B. Garay Gustavo M. Calixto Alexandre M. De Oliveira Marcelo K. Zuffo University of São Paulo, POLI/USP, São Paulo, SP, Brazil Interdisciplinary Center in Interactive Technologies, CITI/USP, SP, Brazil _____________________________________________________________________________________________

ABSTRACT This paper describes the state of the art of a simplified model for integration of services in mega events. The project context, which was financed by the National Research Council – CNPq, is divided into six major areas or functional groups: Urban Mobility, Tourism, Airports, Security, Energy and Telecommunications. The proposal is described in layers, as an infrastructure model of integration and services evaluation, describing its main layers and interaction processes. The proposal described in this paper is restricted to the telecommunication subproject; however, in the modeling of the scenario for the study case, it was necessary to consider requirements and variables that are common to all subprojects. In the specification process of these requirements, we noticed important processes that interact with some sectors in the city of São Paulo; however, in the mean time, we noticed some failures regarding the integration and collaboration performed by administrative elements of stadiums as a main focus in the _____________________________________________________________________________________________ Manuscript first received/Recebido em: 25/02/2014 Manuscript accepted/Aprovado em: 27/06/2014 Address for correspondence / Endereço para correspondência Jorge Rodolfo Beingolea Garay, is B.Sc. in Computing (2004) Universidad Inca Garcilaso de La Vega. M.Sc. in Electrical Engineering (2007) - University of São Paulo, and PhD. in Electrical Engineering (2012) - University of São Paulo. He is currently a PAD Group Researcher in the Integrated Systems Laboratory (LSI - EPUSP) and of Interdisciplinary Center in Interactive Technologies (CITI) at USP. He has experience in Electrical Engineering, with emphasis on Wireless Sensor Networks, Wireless Communication, Wireless Networks and Pervasive Computing, Computer Architectures, Complex Systems, Internet of Things, Service-Based Architectures, and Cyber-Physical System. E-mail: jorge@lsi.usp.br Gustavo M. Calixto, is B.Sc. in Computer Technology (2004) and M.Sc. for University of Campinas. and PhD student in the University of São Paulo. He is currently a Researcher in Interdisciplinary Center in Interactive Technologies (CITI) at USP, with experience in Electrical Engineering, with emphasis on Digital TV. E-mail: calixto@lsi.usp.br Alexandre M. De Oliveira, is B.Sc. in Electrical Engineering with Computer emphasis - Catholic University of Santos (2008), M.Sc. in Electrical Engineering (2012) - University of São Paulo, and PhD student in the University of São Paulo. He is currently Group PAD Researcher in the Integrated Systems Laboratory (LSI) at EPUSP, with experience in Electrical Engineering, with emphasis on VLSI Design, UWB I-Radar, Timed-array propagation, Microwave and Electromagnetism, and Numerical Methods and simulations for Electromagnetism. E-mail: amanicoba@pad.lsi.usp.br Marcelo Knörich Zuffo, Graduated in Electrical Engineer (1989) M.Sc. in Electrical Engineering (1993) and PhD. in Electrical Engineering (1997) at University of São Paulo. He is full professor at University of Sao Paulo since 2006 at the electronics systems department. He is the chief of R&D activities of the Laboratory for Integrated Systems and of Interdisciplinary Center in Interactive Technologies (CITI) at USP focused on interactive technologies, digital health, high performance computing, virtual reality, graphics computing and visualization. He is member of ACM and of Brazilian Digital Television System Forum since its foundation in 2007. E-mail: mkzuffo@lsi.usp.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

346

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

mega event and part of our case study. Currently, the proposal is in its first version as open software for the evaluation of the services quality and the mega event infrastructure through the use of performance indicators. Keywords: Mega Events; Infrastructure Services; Framework; KPI; Cloud.

1. INTRODUCTION Brazil, in the past few years, has been preparing itself to support worldwide events (Mega Events), such as the FIFA World Cup in 2014 and the Olympics in 2016. Both events have the challenge of providing the necessary infrastructure for the proper conduct of the activities that will be performed. Regarding the FIFA World Cup, in which there is a distribution of the host cities in all regions of the country, the logistics of transportation and the offering of basic information to the participating community (athletes, spectators, employees) are essential. The monitoring of services (the entire infrastructure to ensure the good progress of the event) offered to the participating community can result in the generation of indicators, which, in the future, can contribute to the evaluation of lessons learned and also to the analysis of the eventâ&#x20AC;&#x2122;s legacy. The use of mobile devices for easy access and information sharing can be considered as a concept in the exponential growth phase. Currently, the number of users of these mobile devices is increasing (N.Eagle, 2005), as well as the extent of features that, today, these embedded devices offer. The innovation rate of the functions grows along with the market of applications that allow user location tracking (Rose, 2011), multimedia resource sharing, multi-user games (Hassan, 2010), and access to the Internet and social networks etc. The demand for the use of these applications is increasing, resulting in the need to ensure the access to them considering minimum quality parameters that can be enforced through a set of Key Performance Indicators KPI (Coffery, 2011). The flexibility and ease of access to the resources offered by applications that run on mobile devices allow obtaining information through the user's own satisfaction with the services offered in the host cities of the mega sporting events. To provide access to these applications and online services, mega events should have an IT infrastructure that makes it easier for the spectator to use Internet services through dedicated networks, in some cases with signal distribution via Wi-Fi network and via other paid communication services such as the 3G and 4G networks. The locations for mega events also require that cellular phone carriers install transmission antennas in order to maximize, within the area of the event, the quality of data provided by each carrier to its set of customers. As a result of this infrastructure, an extension of applications and useful data for the different areas, such as security, which includes evacuation procedures and crowd control in mega events, can be shared. Some studies mention the importance of the project and management of large groups, such as urban mobility, presenting a theoretical approach to the life cycle of urban transport models before and after mega sporting events take place (Cheng, 2009a; Cossavelou, 2001). In (Cheng, 2009b), the authors present a set of investments in successful technology for the Beijing Olympics. The study aims to evaluate changes in investment models adopted to contribute to a better development of the technological activities of future Olympic Games. In (Hou, 2006), the study included a model of digital transmission by applying the gatekeeping theory that follows a model of five JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

State of the Art: Integrating Services for Mega Events

347

levels to which the authors do not add additional levels, but they believe that new criteria must be applied to an analysis. Variables are composed of 21 questions and the resulting data is processed using an approach to Structural Equation Modeling - SEM. This paper aims to present the state of the art of a service integration model to mega events. The environment or context of the proposal identifies a division into six subprojects, also called major areas or functional groups. In the telecommunications subproject, in which this paper will be performed, it is described an architecture model of hardware and software to integrate the diversity of services and information, considering a user-centered sharing architecture and also the use of performance indicators to evaluate the service quality provided by the proposed model. The methodology of validation of the proposal consists in the description and implementation of a case study: the 2014 FIFA World Cup in Brazil, which was divided into two scenarios. The first scenario is already in development and the first version of the second scenario is already in use. Among the contributions, it is considered the facility to identify or to determine KPI indicators for the success of services integration model for mega events. The paper is organized in the following way: In Section 1, a brief overview of the main related works is presented. In Section 2, a brief view of the mega events and a brief introduction about the project financed by CNPq is presented. In Section 3, themodel overview is presented. In Section 4, we present a description of the proposed architecture. In Section 5, Operational Architecture. In Section 6, we present a case study, and, finally, in Section 7, the conclusions are presented. 2. A VIEW OF MEGA EVENTS The holding of a Mega Event, whichever their nature, can certainly influence the acceleration of the social infrastructure in progress or encourage a clearer view of the deficiencies of current infrastructure and demand more objective investments. Growing countries, such as Brazil, has become the target of important Mega Events, like the Confederations Cup, the World Cup and, soon, the Olympics, etc. Thus, there is a clear need not only to measure the impact of the growth of social infrastructure and socioeconomic influences, which, in both cases, we could identify as legacies of the Mega Event, but also to create new ways to acquire, manage, share and represent the information of the environment, which results from the interaction with people. Specifically in the context of legacies, financing sources and research encouragement, partnerships are initiated with leading academic institutions in the country to study the â&#x20AC;&#x153;Mega Events and their Legaciesâ&#x20AC;? phenomenon. The project on which this work is developed is financed by CNPq and seeks to identify the Legacies of Mega Events like the World Cup 2014. In relation to the overall goal of this project, we must highlight that it is still early for conclusive opinions about holding these Mega Events, their legacy and the population vision in relation to them. This paper focuses on dividing the context into six sub-projects, which, as mentioned earlier, are described as major areas or functional groups (Urban Mobility, Tourism, Airports, Security, Energy and Telecommunications). It aims to analyze not only the legacy for every major area, but also the level of software JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

348

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

solutions and IT infrastructure in general, to facilitate the integration and communication of the various services provided to viewers inside and outside the sporting environments. The telecommunication subproject proposes and describes a model of hardware architecture and software to integrate the diversity of services and information, with the user as the basis of an architecture for sharing. It is also considered the use of performance indicators to assess the quality of services provided. For the development of the proposal, various parameters have been considered to be studied, for example, the high heterogeneity of embedded hardware and communication (Garay, 2013a), protocols devices and the formalization of physical events (Garay, 2013b) that can exist within the context of the proposal. 3. MODEL OVERVIEW The model aims to explore the features and functionalities of mobile devices (mobile phones, tablets, iPads etc.) to obtain information related to major areas or functional groups mentioned in sections I and II. This information can be resulted from activities related to air transport, safety in public environments (stadiums, museums, parks, subway stations etc.), fire alarms, people location tracking etc., all through an infrastructure of massive sharing of information (Murat, 2010) that can be restricted and subsequently separated as services. In the context of the proposal, information on each functional group is managed by applications and treated on the level of integration architecture as a set of services, in which each service corresponds to a proper functional context, which has a set of functional capabilities related to this context, for example, safety related to people location tracking inside the stadium, fire control, lighting systems (energy), public transportation, localization and quality of data communication services etc. The capabilities of the set of services must be appropriated to be invoked by external programs (). INFORMATION CONTEXT Security System connection

USER MOBILE Validation connecion

Traffic Control System

Modeling for System Integration

Communication Data System

External system Events

EXTERNAL SYSTEM

Figure 1: DFD of the proposal. It presents a simplified diagram, however, not less objective, showing the process flow of the proposed architecture, which is described in Section IV. The flow describes four fundamental processes that, according to the extent of modeling, can characterize the complexity of implementation and relevance of each process.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

State of the Art: Integrating Services for Mega Events

349

• Mobile User: It is the term provided to the user set and mobile device. There is, nowadays, a wide range of mobile devices and, as a consequence, the growth of software applications developed is increasing, both for performing simple tasks, like reading, editing, or sharing media (photos, videos), and for controlling automation systems and acquiring context information (real-time and remote). Mobile applications should consider the optimization of some features such as energy (Foll, 2012), when, for example, users take benefit from multiuser applications, such as games, while running some services in the background for acquisition of context information, communication (connection, upload, download ) etc. (Leikas, 2006). The Mobile User is seen as a process that needs to be abstracted at the level of communication (Wi-Fi, 3G, 4G) and in terms of application processes, being used to make use of a data service or to share information within the same context. • Modeling System Integration: The creation of functional and structural models and the implementation of knowledge and control data sources (Hsu, 1990) is considered through an approach to manage spatial information (location tracking, time, attribute). This includes metadata templates and a framework (Luo-Yingwei, 2003) to facilitate the interoperability of processes that result from the integration of existing applications between mobile devices and applications that comprise of or result from the organization of events that correspond to each functional area or group. Service composition techniques (Wan, 2008; Yun, 2013) and events monitoring (Feenstra, 2009) must be considered in the model of system integration, which, as mentioned initially, can act as a framework or as middleware that, in case of seeing the system activities through an interface, can act as a supervisor system. • Context Information: They are the environment information. Currently, environments in which there is a wide availability of mobile devices offer new opportunities to users of these devices to dynamically access resources, information, and services available in the environment (context). Access to these resources, services, data etc. is found by users based on context information characterizing their status and assuming that this information is highly reliable. In the context in which it accesses or shares a service or feature, the quality of context information plays an important role in improving the offered services (Wei, 2010) and ensuring the correct behavior of applications. In applications for mega events, there are several quality parameters, techniques for information sharing to maintain the safety (Qingsheng, 2007), as well as frameworks and middleware which are needed to integrate applications, services and resources with external systems (Yongkay, 2009), all based on information acquired from the context. • External System: Context information is needed for the reliable sharing of information and verification of services which are offered during mega events. The resulting set of information is useful to trigger what we call external systems. External systems are applications that correspond to the various departments responsible for the monitoring of mega events. These dependencies can be local, state, federal related to control of mega events in such aspects as security, traffic control, crowd control, fire, energy etc. The information provided by the user or user group, through a graphical interface, can be classified using semantic techniques along with the creation of ontologies (Key-Sun, 2007) for better classification and representation.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

350

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

During the generation of events to trigger external systems there may be a probability that one event is repeated several times under the same conditions that would make access and the external system itself more proactive. The probability can be represented and calculated using the equation (Eq.1): (1)

Where P(A) is the probability of (A) event to happen, the number of events is A, and N is the total number of possible events. To specifically determine the probability of the same event to happen, in order to predict certain events, we must calculate N according to the characteristics and type of event.

4. ARCHITECTURE DESCRIPTION In the proposed model, applications must operate independently due to service architecture with low coupling favoring high application availability and easy maintenance. Each application environment must operate in their own infrastructure, share and integrate information through a distributed process to solve problems related to reliability and availability of services and applications. In Figure 2, the diagram of the integration model is presented, in which the main layers are described objectively here: • Security Application: This is the first of three main modules implemented at the level of the user interface. In this module, non-functional requirements are considered, such as integrity, availability, and reliability of information for communication with external systems and with user-centered applications. • Communication Application: This is the second module of the user interface level. Here, the logical model of communication is implemented between the components and the framework, and the physical communication model (infrastructure) between applications, services, and end user. • Monitoring Application: This is the last of the three modules at the level of user interface and it is in this module that the model is validated through the case study. Non-functional requirements are defined in this module, as well as the modeling of user interface, which includes performance indicators (KPI) of the services that are supported by the integration model. • Requirements: Here, the extension of functional, non-functional, system, and user requirements are defined, modeled and parameterized. The behavior of requirements is defined by the rules of business of each of the applications that will take advantage of the integration model. In the integration layer shown in Figure 2, we present other processes that control activities and information framework and define integration processes to be considered in the implementation of an application, the integration rules and the interfaces.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

State of the Art: Integrating Services for Mega Events

351

• Integration Framework: In this layer, the model of integration is defined by using a design pattern for implementation, component specification, coupling levels, component scalability, dependencies, and, finally, components reusability. • Service Composed: Commonly, the composition is the reason for something to be decomposed. Something bigger is divided into smaller parts because we identify the potential benefit of being able to do things with these parts that could not be possible if they only existed together. These small parts can be called processes, which we classify into three types: AS (Abstraction Service), CS (Composition Service), and TS (Translate Service). • Interfaces: They must be developed as a result of the application features and customer needs, allowing interaction between framework, applications, and end user. The user-level interfaces are developed for the Android OS. Integration Model Urban Mobility

Application

MV-1

MV-3

MV-2

Security

Airports

Tourism

Energy

Service Resource Management Service Composed

Telecommunication

Cloud Management Node Area (USP)

Security Application Interface Mobile Service Cloud

User Mobile

Figure 2: Diagram of the integration model (Framework). 5. OPERATIONAL ARCHITECTURE In this section, we describe the behavior of the layers of the proposed model and the processes involved in the interaction to allow integration, starting from the context information represented by the Functional Groups. The flow behavior of the model is shown in Figure 3 and Figure 4. In the process of running the operating architecture of the proposed model for the integration of resources, information, and services in mega events, the system performs a mapping from the set of available applications for each functional group, beginning from the communications major area. From a simplified view, shown in Figure 3, each functional group implements, or can possibly implement, a small or large set of applications and services, which, in turn, can trigger external systems. These applications are integrated into the framework by following a process with predefined business rules. The user will be able to access JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

352

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

applications via mobile devices using the IMEI (International Mobile Equipment Identity), of their device as the only identifier. Application

Information Framework

Application details

User Mobile

User details

User ID

Define Process for Integration

Interfaces

Requeriments

Integration Framework

Implement Application Functional Groups

Business Rule

Process Frameowrk

Figure 3: DFD - Integration model of a simplified flow of services. In the diagram presented in Figure 4, the data and process flow are presented, adding one more level to Figure 3. The functional groups or major areas are also shown in the diagram, all integrated by the Communications area (also known, in the project, as Telecommunications). During service composition, it is necessary to define when, how, and by whom these services will be reused. Business rules determine the use of components, the framework integration with applications, and the validation of the user with the system through an interface available to their mobile device. Finally, all data resulted from applications that belong to each of the major areas are stored in a common data repository, considering that not all information is immediately useful for the various running processes; however, these can be further considered relevant, treated, and represented by a statistical model to determine probabilities and quality indicators of events, using a larger number of details.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

State of the Art: Integrating Services for Mega Events

Define Process for Integration

Interface Requeriments

353

Process Frameowrk

User Verification

User Mobile

Application details

Integration Framework

Implement Application

Information Framework

User History

Requeriments

Business Rule

User ID

Service Composed

Mobile Application

Urban Mobility

Security

Functional Groups

Services Model Reusable Services Data Repository

Airports

Tourism

Communications

Figure 4: DFD - Detailed flow of service integration model. 6. CASE STUDY The case study for validation of this proposal is based on the description and modeling of a user-centered application. The goal is to supply the necessary tools to provide, to the user, context information that can be used in three processes: The first would be to identify the services provided by the system, the second would be to determine the quality of services purchased, and the third would be to associate information and services to a major area or functional group (Figure 5).

Tourism Airports

ID User Mobile

Interface User Mobile

Host City

Sports Scenery

Language

Quality of services

Urban Mobility

Cloud Repository

Security

Evaluation

Communication

Localization Interface

Framework

Figure 5: Scenario Diagram for Case Study. The logic of development (coding) of the proposed model is quite robust given its complexity and the formalism that is considered to measure the quality of each service or set of services, considering indicators (KPI).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

354

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

In the case study, two scenarios are described for the implementation. Each scenario represents its own level of complexity; some limitations were identified during the stage of requirements and specification gathering. •

Scenario 1:

Crowdsourcing; it is considered that many events happen inside of the stadium, and some of these events are related to security, which, in turn, is related to other factors such as lighting, policing, emergency exits, observation of criminal acts, etc. For this specific context, it is necessary to model and implement an environment of mobile application in which the user, when observing any event (relevant or not), is able to share the information of this observation with other users, the ultimate goal would be to improve the quality of security. The strongest part of the infrastructure model is centered on the user and their ability to share information with other users via their mobile device, primarily in a collaborative environment in which the communication channel would be a wireless network or the Internet itself. We also consider the exploring of the existing IT infrastructure in the stadiums (Mega Event environments), such as Wi-Fi communication and strong infrastructure implemented by the various phone carriers, to optimize the data connection of their set of customers. •

Scenario 2:

Indicators, people and Mega Events make up a concrete relationship. People (domestic or foreign tourists) move through the various cities hosting a Mega Event using air or ground transportation and, when they are in the city, they use services such as public (subway, bus) or private (rental vehicles or taxi) transportation and take advantage of the major tourist locations. In this cycle, they observe important elements, such as traffic conditions, quality of transport, facility for getting around, lodging, street lighting, policing, perception of safety in the city, in the stadium or in any other environment where they are attending the event, and etc. This set of information is relevant to create indicators, not only about the quality of a service, but also to determine the impact of a direct or indirect legacy of the Mega Event. A robust relationship of indicators, location and host cities is modeled for this scenario. This part of the proposal begins by checking in which of the host cities the user is using their mobile device’s GPS. After this observation, we present a set of interfaces that guide the user through a collaborative process, with questions about the quality of infrastructure and services. When the user's location is identified, the application presents airports, bus terminals, subway stations, stadiums and events (games) that will be performed exclusively in that location. An important aspect is that the user does not have access to indicators or services of the mobile application if they are not in one of the host cities of the Mega Event, thus creating a relationship of reliability and integrity of the collected information. The Mega Event needs to be measured always considering their impact on the services offered to the people (tourists and locals). This impact can be extended even to social infrastructure (legacy), a Mega Event’s exclusive infrastructure and infrastructure

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

State of the Art: Integrating Services for Mega Events

355

and services that were accelerated by the occurrence of the Mega Event. To complete this process, it is necessary to use performance indicators. â&#x20AC;˘

Performance Indicators

The KPI is composed of the following indicators: Availability (Di), Opportunity (Opi), Information (Ini), and Reliability (Coi). The weight of each indicator can change according to the services conditions that are offered by each functional group, for example, the weight assigned to the indicator reliability can switch to the services provided within the functional group of urban mobility, where, at certain times, the flow of vehicles can change as a result of having a greater number of people, Di may be lower as well as Opi to access the same means of transport. (2)

Where is the sum of the products of the weight of each indicator for the value of the corresponding indicator is, Wi is the weight or weighted value of each indicator and Ii is the value of each indicator. Under typical conditions (static), Wi values are considered constant according to the historical development of the service, and, in scenarios of special conditions, it can be dynamically calculated using the following expression, which represents a situation of proportional distribution (Eq. 3): (3)

Where to all the weight of the weighting factor there is a correspondent, there is the value of the indicators that is established by the conditions of the special scenario, and the total value of indicators. When considering the four indicators mentioned initially, the expression (Eq. 2) can be represented in (Eq. 4): (4) â&#x20AC;˘

Cluster environment for the Cloud

The hardware that will be used corresponds to anIBM architecture, strictly a Blade Center with two HS22V blades (Xeon 6C X5650 - 2.66 GHz with 144 GB of RAM) and four HS22 blades with support for 96GB RAM for virtualization. An IBM Director Server (Xeon 4C E5620 - 2.40 GHz) and a DS3500 Storage with 19 SAS disk of 600 GB.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

356

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

Computer Node

Computer Node Gigabit – Ethernet Switch Gigabit – Ethernet Switch

Virtualization Node (Storage Node) Virtualization Node (Storage Node)

Synchronization Hardware

Virtualization Node (Storage Node)

Virtualization Node (Storage Node) Acess Workstation

Administration Workstation

Figure 6: Cluster architecture – IBM Cloud. The IBM BladCenter is a very robust architecture and it is available in CITI (Interdisciplinary Center for Interactive Technologies), from the University of São Paulo (USP), which is why CITI has been the ideal choice to meet a high flow of requests and data storage. It also allows the rationalization of resources when less hardware is required, reducing the short running costs and energy use, which is an ideal situation since it is considered that it should meet, after system deployment, a high flow of requests, but for short time periods. The system virtualization must ensure a perfect circle of lower power consumption and performance, helping to ensure that the high data and requests flow (many events that result from the applications developed for each functional group) is not a factor that impacts on the operating system reliability. Virtualization has opened the way to new organizational models, such as cloud computing, allowing services to be easily deployed considering the condition that a continuous monitoring system has been implemented in the background. •

Environment for performing experiments

The environment for performing system tests (Figure 7) is modeled to ensure correct coding and fewer errors in the data structure, and also to meet the requirements of software quality. The test environment will allow us to mainly eliminate loading errors and to reduce the approval of new applications to be integrated in proposed model during the integration of services considering the extension of each area or functional group within the context of mega events, especially considering the high workload and the behavior of the model under normal conditions. The performance test should start by the web monitoring, hardware monitoring, availability, and response time to service requests and catching errors.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

State of the Art: Integrating Services for Mega Events

357

Load Tester

4 Tablet

Data Performance

3 2

Internet

1 Principal Server Cluster - Cloud

Load Generator Smart Phone

Figure 7: Environment for tests. Another parameter that must be analyzed over the model is the scalability. In this context, scalability means the responsiveness of the system in relation to resource demand that is required from it. It is considered that, in mega sporting events, the data flow generated by applications that will be integrated into the framework for later storage in the data structure must be pretty high. 7. CONCLUSIONS The number of researches and projects performed to develop or propose to manage the hardware infrastructure, applications and information derived from the mega events context is limited, usually to the environment in which the event happens. Details on the procedures performed by a framework or middleware in application integration are largely unknown since in most cases the solution is specific and proprietary. This paper conducted a detailed survey of the main contributions in the mega events area and tried to base the model as well as the use of performance indicators. Although the approach seems to be very comprehensive and, at some points, subjective, it is intended that the descriptions of the layers and processes of the model are the point of reference for the development of applications and services for mega events. In the proposal, we intend to present a model that is consistent with the needs of the project financed by the National Council of Technological and Scientific Development - CNPq, with the goal of creating solutions to manage architectures, applications and information flow in mega events. The processes and activities described in each scenario should be considered relevant to hold any mega event and, on this matter, there is a clear tendency to create highly complex applications, favoring the representation of information and control interaction between physical events and applications (Garay, 2010c). New concepts and methods need to be explored aiming to create a single integrated system of information and control events such as security, telecommunications, traffic, urban transport in real time, etc.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

358

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

The first version of the implementation of the second scenario is available on Google Play and its implementation methodology has been accepted for publication in (Garay, 2014d) and the version, in its view of software product in (Garay, 2014e). The application presents important platform features for application of performance indicators that allow the measure of social impact on services social infrastructure in each of the mega event host cities. We must mention the difficulty accessing the information that would enable a more complete pilot solution, which includes the implementation of the first scenario. The difficulty originates from the resistance of the stadium managers, major focus of the mega event, in providing access to technical information. They would have even favored the planning of the nonexistent communication infrastructure of most stadiums hosting the World Cup in 2014. ACKNOWLEDGMENT The authors would like to thanks CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) that offered support for this work by project Encomendas ME/CNPq - Legados e oportunidades Gerados pela Copa do Mundo 2014 under process number 400054/2013-2 and to the Interdisciplinary Center in Interactive Technologies of University of Sao Paulo – (CITIUSP).

REFERENCES Cheng Xiaoduo, "The Interaction Mechanism between Olympics' Venues and Urban System," Management and Service Science, 2009. MASS '09. International Conference on , vol., no., pp.1,4, 20-22 Sept. 2009. Cheng Xiaoduo, "The Urban System Impact on Post-Games Development of the Olympics' Venues in China," Computer Science and Information Technology - Spring Conference, 2009. IACSITSC '09. International Association of , vol., no., pp.397,402, 17-20 April 2009. Coffey, V.; Willar, D.; Trigunarsyah, B., "Quality management system and construction performance," Quality and Reliability (ICQR), 2011 IEEE International Conference on , vol., no., pp.403,407, 14-17 Sept. 2011. doi: 10.1109/ICQR.2011.6031750. Cossiavelou, V.; Bantimaroudis, P.; Kavakli, E.; Illia, L., "Wireless communications in Olympic Games: R&I into the frame of the updated gatekeeping model," Wireless Telecommunications Symposium (WTS), 2011 , vol., no., pp.1,5, 13-15 April 2011. Feenstra, R.; Janssen, M., "Service Composition in Public Networks: Results from a Quasi-Experiment," System Sciences, 2009. HICSS '09. 42nd Hawaii International Conference on , vol., no., pp.1,10, 5-8 Jan. 2009. doi: 10.1109/HICSS.2009.385. Foll, S.; Herrmann, K.; Rothermel, K., "Energy-Efficient Update Protocols for Mobile User Context," Advanced Information Networking and Applications (AINA), 2012 IEEE 26th International Conference on , vol., no., pp.120,127, 26-29 March 2012. doi: 10.1109/AINA.2012.129. Garay, J.R.B.; De Oliveira A. M., Kofuji, S.T., "Proposed Middleware for Sensor Networks in Cyber-Physical System Environments", in Proc. SENSORCOMM 2013 : JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

State of the Art: Integrating Services for Mega Events

359

The Seventh International Conference on Sensor Technologies and Applications. pp. 178-185, 2013. Garay, J. R.B; De Oliveira A. M. Kofuji, S.T. , "Cyber-Physical Systems (CPS): Description of a Solution to Heterogeneous Architectures Management", in Proc. LATINCOM 2013: 5th IEEE Latin-American Conference on Communications 2013. Garay, J.R.B.; Kofuji, S.T., "Architecture for sensor networks in cyber-physical system," Communications (LATINCOM), 2010 IEEE Latin-American Conference on , vol., no., pp.1,6, 15-17 Sept. 2010. doi: 10.1109/LATINCOM.2010.5641126 Garay, J.R.B, Calixto G. M, De Oliveira A. M., Zuffo M. K." Performance Indicators: A Mobile Solution to Identify Legacies in Mega Events", The 4th IEEE International Conference on Consumer Electronics - Berlin (IEEE-ICCE-Berlin). 2014. Garay, J.R.B, Calixto G. M, De Oliveira A. M., Zuffo M. K." Mobile Applications and Legacies of Mega Events", in Proc. INTERCON 2014: International Congress in Electronic, Electrical and Engineering. vol. 21 pp. 315-319. PerĂş. 2014. Hasan, S.I.; , "Multiplayer gaming for low-end mobile phones: Gaming between basic mobile phones, handheld devices and computer platforms," Games Innovations Conference (ICE-GIC), 2010 International IEEE Consumer Electronics Society's , vol., no., pp.1-5, 21-23 Dec. 2010. Hou Yu-peng; Ju Xiao-feng, "The Study on the Influence Channel and the Degree of Impact of Olympic Economy of Beijing," Management Science and Engineering, 2006. ICMSE '06. 2006 International Conference on , vol., no., pp.1145,1150, 5-7 Oct. 2006. doi: 10.1109/ICMSE.2006.314204 Hsu, C.; Bouziane, M.; Cheung, W.; Nogues, J.; Rattner, L.; Yee, L., "A metadata system for information modeling and integration," Systems Integration, 1990. Systems Integration '90., Proceedings of the First International Conference on , vol., no., pp.616,624, 23-26 Apr 1990 doi: 10.1109/ICSI.1990.138727 Key-Sun Choi, "IT Ontology and Semantic Technology," Natural Language Processing and Knowledge Engineering, 2007. NLP-KE 2007. International Conference on , vol., no., pp.14,15, Aug. 30 2007-Sept. 1 2007. doi: 10.1109/NLPKE.2007.4368004 Leikas, J.; Stromberg, H.; Ikonen, V.; Suomela, R.; Heinila, J., "Multi-user mobile applications and a public display: novel ways for social interaction," Pervasive Computing and Communications, 2006. PerCom 2006. Fourth Annual IEEE International Conference on , vol., no., pp.5 pp.,70, 13-17 March 2006. doi: 10.1109/PERCOM.2006.38 LuoYingwei; Wang Xiaolin; Xuzhuoqun, "Spatial semantic network and agent-based framework for spatial information interoperation," Information Reuse and Integration, 2003. IRI 2003. IEEE International Conference on , vol., no., pp.40,47, 27-29 Oct. 2003. doi: 10.1109/IRI.2003.1251393 Murat Demirbas, Murat Ali Bayir, CuneytGurcanAkcora, YavuzSelimYilmaz, and HakanFerhatosmanoglu. 2010. Crowd-sourced sensing and collaboration using twitter. InProceedings of the 2010 IEEE International Symposium on A World of Wireless, Mobile and Multimedia Networks (WOWMOM '10). IEEE Computer Society, Washington, DC, USA, 1-9. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

360

Garay, J.R.B., Calixto, G.M., De Oliveira, A.M, Zuffo, M.K.

N. Eagle and A. Pentland, “Social serendipity: Mobilizing social software,” IEEE Pervasive Computing, vol. 04-2, pp. 28–34, 2005. Qingsheng Zhang; Yong Qi; Jizhong Zhao; Di Hou; Tianhai Zhao; Liang Liu, "A Study on Context-aware Privacy Protection for Personal Information," Computer Communications and Networks, 2007. ICCCN 2007. Proceedings of 16th International Conference on , vol., no., pp.1351,1358, 13-16 Aug. 2007. doi: 10.1109/ICCCN.2007.4318009 Rose, R.; Meier, C.; Zorn, S.; Goetz, A.; Weigel, R.; , "A GSM-network for mobile phone localization in disaster scenarios," Microwave Conference (GeMIC), 2011 German , vol., no., pp.1-4, 14-16 March 2011. Wan Fang; Shen Jianjing; Han Ding; Zhang Hui, "On composition of service component based on service-oriented architecture," Control Conference, 2008. CCC 2008. 27th Chinese , vol., no., pp.573,576, 16-18 July 2008. doi: 10.1109/CHICC.2008.4605721 Wei Zhang; Chang, C.K.; Taiming Feng; Hsin-yi Jiang, "QoS-Based Dynamic Web Service Composition with Ant Colony Optimization," Computer Software and Applications Conference (COMPSAC), 2010 IEEE 34th Annual , vol., no., pp.493,502, 19-23 July 2010 doi: 10.1109/COMPSAC.2010.76 Yongkai Zhan; Shuangquan Wang; Zhuang Zhao; Canfeng Chen; Jian Ma, "A mobile device oriented framework for context information management," Information, Computing and Telecommunication, 2009. YC-ICT '09. IEEE Youth Conference on , vol., no., pp.150,153, 20-21 Sept. 2009. doi: 10.1109/YCICT.2009.5382404 Yun Ma; Xuanzhe Liu; Yihan Wu; Grace, P., "Model-Based Management of Service Composition," Service Oriented System Engineering (SOSE), 2013 IEEE 7th International Symposium on , vol., no., pp.103,112, 25-28 March 2013. doi: 10.1109/SOSE.2013.21

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.345-360

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 361-378 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200008

USING THE BSC FOR STRATEGIC PLANNING OF IT (INFORMATION TECHNOLOGY) IN BRAZILIAN ORGANIZATIONS Adriano Olímpio Tonelli Instituto Federal de Educação, Ciência e Tecnologia de Minas Gerais, Formiga, MG, Brasil Paulo Henrique de Souza Bermejo André Luiz Zambalde Universidade Federal de Lavras, Lavras/MG, Brasil _____________________________________________________________________ ABSTRACT This paper presents a method that integrates Balanced Scorecard (BSC) concepts and IT strategic planning (ITSP) processes. The resulting method was applied in two organizations in order to verify contributions of Balanced Scorecard regarding the identified ITSP problems. The development of this work was realized in qualitative, exploratory research based on two case studies. The results show that the use of BSC contributed directly to work with IT strategic planning challenges and involved middle management, multifunctional teams, and top management support, all beyond IT boundaries. Keywords: information technology strategic planning; balanced scorecard; information system management; IT strategic management; IT strategic alignment.

_____________________________________________________________________________________________ Manuscript first received/Recebido em: 17/05/2012 Manuscript accepted/Aprovado em: 05/02/2014 Address for correspondence / Endereço para correspondência Adriano Olímpio Tonelli, Professor do Instituto Federal de Educação, Ciência e Tecnologia de Minas gerais, doutorando em Administração pela Universidade Federal de Lavras (UFLA). Possui mestrado em Administração, graduação em Ciência da Computação e MBA em Governança de TI, pela UFLA. Atua em atividades de pesquisa nas áreas de sistemas de informação, inovação, desenvolvimento de software e gestão estratégica de TI. E-mail: tonelli@dcc.ufla.br Paulo Henrique S. Bermejo, professor adjunto do Departamento de Ciência da Computação da Universidade Federal de Lavras (UFLA). Formado na área de computação, Paulo Henrique realizou pós-doutorado na Universidade de Bentley (EUA), e doutorado em Engenharia e Gestão do Conhecimento pela Universidade Federal de Santa Catarina. Atua em atividades de pesquisa e ensino nas páreas de sistemas de informação, inovação e gestão estratégica de TI. E-mail: bermejo@dcc.ufla.br André Luiz Zambalde, Professor Associado da Universidade Federal de Lavras-MG (UFLA), nos Departamentos de Ciência da Computação e Administração. É Engenheiro de Telecomunicações (INATEL-MG 1984). Pós-graduado em Administração (UFLA-MG 1989), Mestre em Eletrônica (UNIFEI-MG 1991), Doutor em Engenharia de Sistemas e Computação (COPPE/UFRJ 2000), com Pos-Doutorado em Ciência da Computação (UFMG-MG 2005) e Estatística e Gestão da Informação (UNINOVA - Portugal PT 2010). E-mail: zamba@dcc.ufla.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

362 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

1. INTRODUCTION The strategic use of IT resources has recently attracted increased interest from researchers and organizations. Information technology (IT) has been established in the business world as a fundamental factor for corporate strategy development and realization (Gutierrez & Serrano, 2008). However, ensuring that IT investments and initiatives deliver better performance and value to corporate strategies has become the main challenge for IT and business executives (Gutierrez & Serrano, 2008; Lee & Bai, 2003). Based on this tendency in information technology in organizations, researchers (Grover & Segars, 2005; Lederer & Salmela, 1996; Lutchen, 2004; Mentzas, 1997; Min, Suh, & Kim, 1999) have proposed and applied IT strategic planning methods. The adoption of ITSP methods and techniques has provided many benefits for organizations. These methods and techniques provide ways to develop a planning structure, improve the relationships between business and IT, support an identification of risks and threats, and allow strategies for evaluation and review (Cerpa & Verner, 1998). ITSP methods are essential for aligning IT and organizational strategies in a way that enhances competitive advantages, creates market opportunities, provides direction, and concentrates efforts (Boar, 1994; Lee & Pai, 2003; Newkirk, L., & Srinivasan, 2003; Pai, 2006). However, in practice, promoting the effectiveness of IT strategic planning has been a major challenge to executives (Bechor, Neumann, Zviran, Chanan, & 2010;(Lee & Bai, 2003). As a result, IT strategic planning processes have faced many problems. According to Boar (1994), IT has historically made strategic plans that are either completely or partially devoid of respect within these business goals or with general organizational contexts. Previous surveys have found that more than half of all ITSP-involved stakeholders are not satisfied with the results (Bechor, et al., 2010). The proliferation of the web and internet as business platforms, the outsourcing of IT resources and services, service-oriented architectures, cloud computing, and IT pervasiveness tend to expand ITSP development activities, which then creates new challenges to IT strategic planning (Grover & Segars, 2005). Many other challenges and deficiencies associated with ITSP development have been identified in the literature. Major challenges associated with IT strategic planning include stiffness of the applied methods (Lederer & Salmela, 1996; Salmela & Spil, 2002), difficulty in applying complex methodology (Grover & Segars, 2005; Lederer & Sethi, 2003; Salmela & Spil, 2002; Segars & Grover, 1998), a lack of support from top management on ITSP execution and IT strategic implementation (Lederer & Sethi, 2003), excessive focus on IT functions inside specific aspects (Heckman, 2003), appropriated involvement of middle-level managers (Heckman, 2003), low participation of multifunctional teams (Earl, 2003; Huang & Hu, 2007), distance and low relationship levels between formulation process (planning) and strategies implementation (Broady-Preston & Hayward, 1998; Lederer & Sethi, 2003; Littler, Aisthorpe, Hudson, & Keasey, 2000), and a lack of learning and knowledge emphasis (Bermejo, Tonelli, Brito, & Zambalde, in press; Lederer & Salmela, 1996; Lee & Pai, 2003; Pai, 2006).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

Using the BSC for Strategic Planning of IT in Brazilian Organizations

363

As IT departments have faced these challenges to develop the ITSP, Balanced Scorecard (Kaplan & Norton, 1997) has been changing and consolidating as an important tool for strategic management (Kaplan & Norton, 2008), including IT strategy development and implementation (Huang & Hu, 2007; Simon, 2011). Keyes (2005) highlights that, to promote a better alignment, a BSC must be intimately attached to an IT strategic planning process. This integration must extend beyond the simple capability of balanced scorecard in measuring improvements in IT. Despite the importance attached to ITSP development (Keyes, 2005), the Balanced Scorecard has not been extensively explored in IT strategic planning methods currently available in the literature. Balanced scorecard applications in IT function include BSC utilization as a mechanism to support the implementation of IT strategies (Littler, et al., 2000), promote IT-business strategic alignment and better communication (Huang & Hu, 2007), and conduct IT performance analysis (Velcu, 2010). In addition, other works focus on personalized structure development for IT balanced scorecards (Van Grembergen & De Haes, 2005) and general IT BSC development orientations (Niven, 2002). Studies that consider Balanced Scorecard application for short-, medium-, and long-term planning, including IT strategy formulations, have not been addressed in the literature. Additionally, in the broader context of ITSP studies, according to Grover and Segars (2005), only a few studies have emphasized questions about how to develop ITSP from a BSC perspective through a defined process, including planning process comprehension and whether it provides satisfactory results for an organization. This work aims to investigate the integration between ITSP and BSC in order to mitigate problems in IT strategic planning initiatives. Of these problems, we highlight the lack of support from top management (Lederer & Sethi, 2003), excessive focus on specific aspects of IT functions (Earl, 2003), the need for ITSP expansion over frontiers (Grover & Segars, 2005), low involvement of middle and top management (Lederer & Sethi, 2003), and infrequent use of multidisciplinary teams (Earl, 2003). This paper is organized as follows. Section 2 includes the paperâ&#x20AC;&#x2122;s methodology. Section 3 shows the ITSP-BSC method. Section 4 summarizes the results obtained with the application of the proposed method in two organizations and then discusses the findings of this application. Finally, Section 5 describes conclusions, research limitations, and proposals for future works. 2. METHODOLOGY This study integrates the concepts of Balanced Scorecard (Kaplan & Norton, 1997; Keyes, 2005; Niven, 2002; Van Grembergen & De Haes, 2005) and IT strategic planning (Bechor, et al., 2010; Bermejo, et al., 2012; Boar, 1994; Broady-Preston & Hayward, 1998; Earl, 2003; Grover & Segars, 2005; Lederer & Sethi, 2003; Lee & Bai, 2003; Littler, et al., 2000; Newkirk, et al., 2003; Pai, 2006; Salmela & Spil, 2002; Segars & Grover, 1998). The integration between BSC and ITSP was obtained using a deductive manner (Gimbel, 1968), which resulted in an ITSP-BSC method. This method was used to empirically examine two organizations, and the examination was based on qualitative data collected during the implementation of the proposed method. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

364 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

Therefore, the research can be characterized as exploratory, based on qualitative and longitudinal data, and conducted within a case study approach (Jung, 2004; Yin, 2005). The data was collected from a semi-structured questionnaire, participant observation, and focus groups. These activities were performed by student teams from a graduate program of business education who were trained, oriented, and led by the authors. The selection of the organizations was judgmental (Jung, 2004). The criteria followed the organizationâ&#x20AC;&#x2122;s top management interests and their availability in providing conditions for the case study implementation. In addition, we selected organizations that have a prior corporate BSC. 3. THE REFERENCE ITSP METHOD To develop the proposed method, specific studies about ITSP, IT Governance and Management and Balanced Scorecard were considered. The ITSP studies considered were Boar (1994), Lederer and Salmela (1996), Lutchen (2004), and Bermejo et al. (2012). The ITSP phases proposed by Bermejo et al. (2012) were used as a reference to the structure of the proposed method. This choice was made considering that the method proposed in Bermejo et al. (2012) is aligned with IT Governance features and practices and shows a systematic structure that made possible to incorporate BSC practices in ITSP. In addition, the IT governance framework of Cobit (ITGI, 2007) was utilized. The proposed method is structured in five phases: (1) IT alignment to business; (2) capacity and performance evaluation; (3) IT strategic planning; (4) IT tactic planning; and (5) ITSP results on socialization and closure. In phase 1, IT visions and IT goals are defined for the organization. In phase 2, IT performance and capability for addressing the defined IT goals are evaluated. In this phase, the IT resources and critical processes are identified and analyzed. Conducting phases 1 and 2 makes it possible to identify gaps between current IT capabilities and future directions for strategic IT use in the organization. In phase 3, strategic actions are defined to narrow the gaps between current capabilities and future directions. In this phase, performance indicators are also developed to measure the performance of IT strategic actions. In phase 4, IT tactical projects are developed to implement the IT strategic actions defined in phase 3. In this phase, an IT strategic portfolio is developed. Finally, in phase 5, the results are shared with the stakeholders. BSC application in the ITSP method The following BSC concepts and practices were incorporated into the previously presented IT strategic planning process: 1) balanced perspectives to translate IT vision into IT goals and indicators; 2) strategic maps to establish cause-and-effect relationships among IT goals; and 3) performance vector indicators and results indicators. Additionally, alternative concepts and perspectives proposed for BSC application for information technology were used. Balanced perspectives developed by Van Grembergen and De Haes (2005) for an IT BSCâ&#x20AC;&#x201D;corporate contribution, user, JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

Using the BSC for Strategic Planning of IT in Brazilian Organizations

365

operational excellence, and future orientation—were incorporated as an alternative to Kaplan and Norton’s (1997) traditional perspectives—financial, customer, internal, and learning and growth. An IT BSC cascade was also incorporated during phase 1 to define the IT BSC aligned with corporate strategies. Table 1 shows results provided for IT BSC incorporation along method phases. Table 1. IT BSC evolution through the proposed method Phase 1 2 3

4 5

BSC-related activities incorporated into the ITSP method IT BSC initial development; IT vision and goals definition according to different perspectives in an IT BSC SWOT analysis and analysis of IT critical processes’ maturity, based on thematic defined by the balanced perspectives in the IT BSC IT BSC increment; inclusion of strategic actions and performance indicators development Strategic maps development: inclusion of cause and effect relations between IT goals and performance vector and result IT BSC increment; inclusion of projects and strategic services development for IT strategic portfolio composition IT BSC release to the organization

During the first phase, activities were carried out in order to develop an initial IT BSC structure. These processes were based on the business goals, corporate BSC, and indicators to translate an IT vision into different relevant perspectives that could then be related by IT. An IT vision is represented in different perspectives. It is presupposed that the use of balanced perspectives to describe IT goals can1) direct ITSP activities over IT frontiers; 2) encourage top management and business unit members’ involvement and 3) create the bases for multifunctional teams’ development for ITSP conduction. Through the use of balanced perspectives, it is necessary to identify and analyze strategic IT demands from different organization units, including financial and billing issues, customer relationships, and internal processes in order to support continuous business growth and sustainability. In phase 2, the balanced perspectives defined in phase 1 are used to drive the points to be considered in the evaluation of IT strategic questions, considering aspects inside and outside the organization. It is presupposed that the perspectives incorporated in phase 1 can direct the analysis of IT concerns in different organizational units, thus involving1) top management members to provide information relevant to IT capacity in attending costs, budget, and revenue goals; 2) organizational unit members—IT customers and users—to provide demands for strategic IT support to critical business processes; and 3) IT staff to demand IT internal capacity in terms of staff, IT service delivery, and support. In phase 3, strategic indicators and strategic maps are included in the BSC. In phase 4, an IT portfolio is included. Finally, in phase 5, the IT BSC is released to the organization.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

366 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

Through the incorporation of strategies and indicators organized on strategic maps, it is presupposed that the proposed method can contribute to a better synergy between IT and other business strategic units. This assumption is based on the potential of strategic maps to establish dependency between IT internal issues (internal perspective), sustainability and capacitating (learning and growing perspective), customers’ IT demands (customer perspective), and owners and or shareholders (financial perspective). 4. APPLYING THE PROPOSED METHOD The method was applied to two organizations from different sectors, one from the pharmaceutical sector and another from the health sector. The empirical results will be presented by summarizing the results of each phase of the method in each organization studied. Organization Alpha Organization Alpha operates in the pharmaceutical sector. The organization is 45 years old and has a portfolio of 240 products. In the 2000s, the organization adopted expansion for generic drug lines as the main direction for the business. Based on this vision, the organization’s focus turned to the creation of productive expansion, aiming to facilitate the increase of products´ quality. Phase 1: IT-business alignment. Through the method application in Organization Alpha, it was possible to develop, during phase 1, the IT BSC initial structure with an IT vision related to efficient responsiveness and business growth support. During phase 1, which involved the strategic committee, commercial director, industrial director, and administrative director, it was possible to define1) the strategic orientation for business expansion aligned to increased quality in the products, and2) specific directions related to planning, production, and quality control. Therefore, production and quality control were defined as the two areas in which IT needed to provide strategic support. Table 2 illustrates the involvement of stakeholders—top management, strategic areas and IT managers—at the development of the IT BSC initial version, as well as the results obtained from this involvement.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

Using the BSC for Strategic Planning of IT in Brazilian Organizations

367

Table 2. Stakeholdersâ&#x20AC;&#x2122; involvement on IT BSC initial version development in Organization Alpha Event Startup meeting

IT vision IT goals

Stakeholders involved IT + top management

Top management + strategic areas + IT Top management + IT Organizational areas + IT

Organizational areas + IT

Results Strategic priority: business expansion and increase in quality Strategic areas: planning, production, and quality control To become a responsive sector to support business expansion. Financial perspective: Efficiency in IT expenditures and support to business expansion and product quality Customer perspective: Guarantee responsiveness to production and quality control demands relative to raw material sampling and weighing Improve quality control through IT solutions Internal perspective: 4. Optimize IT services delivery and IT infrastructure Learning and growth perspective: 5. IT capacitating in strategic competences: production and quality

Based on the IT vision, it was possible to define the IT goals in different balanced scorecard perspectives. Phase 2: Capacity and performance analysis. Based on the initial BSC structure, the following themes were defined to guide the capacity and performance analysis: 1) cost controlling process (financial perspective); 2) IT services in attending to quality control and production processes (customersâ&#x20AC;&#x2122; perspective); 3) IT internal organization (internal perspective); 4) IT teamsâ&#x20AC;&#x2122; competence in production process, quality control, and financial management, and 5) top management competence on IT demand identification and analysis (learning and growth perspective). Table 3 shows a sample of the SWOT analysis conducted in Organization Alpha, considering the main weaknesses found in the data collection process. Due to space limitations, we present only the most representative parts of the SWOT analysis, that is, the problems that need to be overcome in order to strategically direct IT.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

368 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

Table 3. Synthesis of Organization Alpha SWOT analysis results Theme Cost control

Service quality

Internal Organization

Competences and communication

Perspective Financial

Weaknesses Costs allocation is done by IT leader, without alignment with strategic business priority Costs allocation do not reflect IT expansion and flexibility strategic priorities Processes for monitoring costs and returns are absent Customers Low understanding of how IT can be involved in changes within organizational units Deficiencies in defining priorities for changes according to business strategies Low involvement of customer areas during definitions of scope and quality requirements in IT projects High incidences of rework on projects due to inaccurate scope specification Internal Unclear definition of responsibilities within IT function Reactive approaches in treating IT incidents Lack of information about IT infrastructure due to an ad-hoc configuration management Learning and Minimal knowledge of IT relating growth to strategic priorities for effort directing IT staff lacks clear knowledge of the strategic priorities of the business

Involvement Commercial Director Administrative Director Industrial Director IT Manager Quality Manager Planning and Production Manager IT Manager

IT Manager

IT Manager Quality Manager Planning and Production Manager

Phase 3. IT strategic planning. Considering IT goals to be achieved (phase 1) and major deficiencies of IT to such achievement (phase 2), phase 3 execution defined strategic maps for IT followed by performance actions and indicators. The IT strategic map for Organization Alpha starts from the learning and growth perspective in order to provide conditions for the proper development and execution of critical processes that will deliver value to the business units. Once effectively and efficiently executed, these processes could ensure responsiveness to production and quality control demands that, in turn, will support top management demands for expansion and cost control.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

369

Using the BSC for Strategic Planning of IT in Brazilian Organizations

Figure

illustrates

the

strategic

map

developed

for

Organization

Alpha. Strategic actions

Finantial

1. Make IT costs efficients and support expansion and billing

Custumers

Objectives

2. Ensure responsiveness to production demands about sampling and weighing of raw materials

· ·

Develop management process of IT investments; Close relationship between IT strategic committee with the process of managing IT investments. Implement reverse charge mechanism

5. Deliver projects on time, budget and according to quality standards

Learning and Growth

Internal

4. Optimize IT services delivery infrastructure

5.IT Training on strategic skills: production processes. Quality processes and financial management

Modification of project management process for continuous interaction of IT teams and business units Creation of IT strategy committee to identify demands and prioritize projects

· · · · ·

Service desk implementation Development of configuration management process Development of change management process Incorporate quality management to project management process Focus on shared IT services

· ·

Training: development of cross-training program Protmote co-location of IT, production and quality staffs.

Figure 1. IT strategic map of Alpha Organization Following the creation of the strategic map, indicators associated with IT goals were developed. Figure 2 illustrates the set of indicators and cause-effect relationship developed in the organization. For confidentiality purposes, we do not include the financial perspective. Internal 4. Optimize delivery infrastructure of IT services Measures Indicators

Learning and Growth

Measures

Customers

Number of shared services

2. Ensure responsiveness to production demands on what is related to sampling and weighing of raw materials

Alignment of shared services to customer demands

Direct

5. IT Training in strategic skills: production processes. Quality processes and financial management

Percentage of conducted changes in accordance with predefined procedures

Measures

Indicators

Number of IT collaborators moved to strategic sectors

Indicators

Changes percentage that attend deadlines and quality criteria

Direct

Number of training hours per colaborator per area of strategic competence

5. Deliver projects on time, budget and according to quality standards Measures

Percentage of business initiatives with delay due to IT; Percentage of business initiatives without IT involvement in early stages;

Indicators Number of quality assurance activities incorporated per project Percentage of delivered projects within strategic areas quality criterias

Percentage of projects that incorporate procedures for interaction between business teams and IT Production downtime due to IT failures

Figure 2.Organization Alpha strategic map and IT indicators

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

370 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

Based on the results obtained in phase 3, synergy between IT and other areas in the organization were identified. In Organization Alpha, synergies were raised between IT function, production and quality sectors, and top management. Table 4 describes the synergies defined in Organization Alpha. Table 4. Synergies defined on Organization Alpha Synergy Description IT + Joint development of training program Production Identification of services that are candidate for shared IT services Composition of IT project team for Production Sector Interaction by strategic committee IT + Top Interaction by strategic committee Management Announcement of new strategic priorities for cost allocation and investments prioritization

Related perspective Learning and Growth Internal Customers Customers Financial

These interaction profiles were considered in phase 4 when projects for tactical plan composition were defined. Organization Beta Organization Beta, a fictitious name applied for confidentiality, acts in the private health care sector. Business operations in the organization are strongly regulated by the Health National Agency, especially with regard to pricing, accounting, and how customersâ&#x20AC;&#x2122; health information is handled. Phase 1: IT alignment to business. During the execution of phase 1, the following stakeholders were involved: 1) CEO, CFO, and juridical departments, to support definitions in financial perspectives and to make appointments of strategic areas in the organization; 2) general management, exchange and proceedings area, and accounting (customer perspective); and 3) information technology management (internal perspectives and learning and growth). Table 5 illustrates BSC initial development in Organization Beta, considering the involvement of different stakeholders.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

Using the BSC for Strategic Planning of IT in Brazilian Organizations

371

Table 5. IT initial BSC in Organization Beta Event Startup meeting

ITVision

IT objectives

Involvement IT + Top Management

Results Main focus: Ensure compliance with regulatory requirements in the health sector Prioritize strategic areas for compliance: general management, juridical sector, and exchange and authorizing Top Management Ensure conformity of IT with regulations + Strategic Areas regarding health information protection + IT Top Management Financial perspective: + IT Minimize financial impacts arising from service interruption and related penalties Reduce financial losses arising from service interruption Exchange and Customers perspectives: Authorizing + IT 3. Ensure protection of sensitive information related to authorization of health services and accounting 4. Promote awareness of and training on health information security IT + Strategic Internal perspective: areas 5. Ensure, through web services and IT management practices, security in transactions involving health information 6. Ensure continuity of systems that support transactions associated with billing and health information processing 7. Create close ties between human resource management and information security management Learning and growth perspective: 8. Improve IT staff competencies associated with IT service delivery. 9. Improve employees’ competencies associated within formation security.

Phase 2: Capacity and performance analysis. In this phase, strengths, weaknesses, opportunities, and threats were gathered, and the following issues were considered: 1) treatment of regulations related to health information protection and services continuity (financial perspective); 2) IT services delivered to authentication, authorization and accounting (customers’ perspective); 3) relationship between IT areas and human resources on what is relevant to information security issues related to people (customers’ perspective); 4) IT internal processes and resources for delivering IT services related to the organization’s strategic areas (internal perspective of learning and growth).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

372 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

Table 6 shows a sample of Organization Beta SWOT analysis results, considering major weaknesses found during data collection. Table 6. Beta Organization SWOT analysis results synthesis Theme Compliance treatment

Perspective Financial

Weaknesses Low understanding about the current status of the organization´s compliance with legal requirements Weak alignment between top management, IT function, and general management to meet new legal requirements IT services Customers Weak interaction between IT delivery function and other organizational units to define IT services levels Lack of mechanisms to monitor IT service delivery to organizational units Relationship Customers Low understanding of HR with HR management on what information must be protected Essential technical focus of IT regarding information security. Internal IT Internal IT leadership with essentially process and Learning and technical capacity and weak resources growth interdisciplinary competencies Reactive approach in dealing with IT service continuity concerns Low interaction with organizational units and top management to discuss and treat new compliance requirements

Involvement Executive Director Financial Director Juridical Department Exchange Authorization Accountability

Human Resources

Information Technology

Phase 3. IT strategic planning Based on IT goals to be achieved (phase 1) and on major IT deficiencies identified (phase 2), in phase 3, the strategic map was defined for Organization Beta. The strategic map developed for Organization Beta starts from the learning and growth perspective to create the necessary conditions—staff capabilities and interaction among organizational units—to conduct internal processes with required levels of information security.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

373

Using the BSC for Strategic Planning of IT in Brazilian Organizations

Figure 3 shows the strategic map developed on Beta Organization. Objectives Minimize financial impacts arising from regulations noncompliance

Strategic actions

Minimize financial losses resulting from interruption of health procedures authorization services

Ensure protection of sensitive information on systems that support authorization, accounting and exchange process;

Systems to ensure continuity of transactions involving billing and health information

Ensure, through use of web services and management practices, safety in transactions involving health information;

Develop closer links with management of human resources for information security management

Training IT staff in service management and IT security

Training of employees in information security procedures.

Develop performance monitoring mechanism in information security and treatment to compliance

Consolidate information security committee : alignment between identification of requirements compliance with HR management and management and processes of IT services Implementing information security management process in HR

· ·

· · · ·

Implement service desk and incident management for security and continuity Implement service level managing process for security and continuity

Creation of information security committee: identification and reporting of compliance requirements for security Consultancy for training IT teams Cross-training. Environment redesign for physical approach of IT with strategic areas.

Figure 3. IT strategic map of Organization Beta Following the creation of strategic maps, indicators associated with IT goals were developed. Figure 4 shows the set of indicators and the cause-effect relationship developed in Organization Beta. Internal

Learning and Growth Promote awareness and collaborator training on health information safety procedures

Percentage of employees with full training in compliance procedures by sector

Indicador Monthly downtime of transaction support system

Ensure, through use of web services and management practices, safety in transactions involving health information;

Finantial Minimize financial impacts arising from regulations non-compliance

Measures

Direct

Indicators Number of compliance incidents caused by custumer area colaborator per month

Direct

Measures

Custumers Ensure protection to sensitive information on systems that support authorization, accounting and exchange process;

Indicators Number of controls implemented by support procedure to sensitive information Number of implemented controls / number of required controls

Measures Indicator Number of implemented controls / number of required controls for web service.

Figure 4. Strategic map and IT indicators on Beta Organization

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

Direct

Ensure continuity of transactions support systems involving billing and health information;

Measures Indicator Number of failures of comply with financial loss

374 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

In phase 3, synergies with different organizational unitsâ&#x20AC;&#x201D;juridical, exchange and authorization of health proceedings, human resources, and top management sectorsâ&#x20AC;&#x201D; were observed. Table 7 shows the synergies observed in this phase. Table 7. Description of synergies defined in Organization Beta Synergy IT + Exchange and Authorization

IT + Juridical IT + HR

Description Development of a training program for information security and IT service level management Interaction for compliance requirement impact analysis Development of IT service levels

Related perspective Learning and growth

Customers Internal

Identification and analysis of Customers compliance requirements Definition and maintenance of Customers information security processes for HR

As in Organization Alpha, these interaction profiles for Organization Beta were considerate in phase 4, when projects to tactical plan were defined. 5. CONCLUSIONS AND FURTHER RESEARCH In this paper, we propose a method that integrates IT strategic planning and Balanced Scorecard concepts in order to provide a systematic approach to direct IT in alignment with strategic business priorities. The application of the proposed method in two organizations showed that the integration between BSC and ITSP can contribute to the expansion of ITSP over IT frontiers and involve both business units and top management. The results show that the definition of balanced perspectives in the method contributed to the identification and involvement of relevant stakeholders that are outside the IT function. Therefore, based on the two case studies, we conclude that the proposed method can be an instrument that contributes to expansion of IT frontiers (Grover & Segars, 2005), and it can mitigate the excessive focus on IT internal aspects and decrease the use of multidisciplinary teams (Earl, 2003). The main contributions of this study are, through the proposed method, the clarification of how to incorporate the Balanced Scorecard in an IT strategic planning effort. Through a presentation and application of an ITSP method that incorporates balanced scorecard concepts, this work provides guidelines to mitigate common IT strategic planning challenges, such as the expansion of ITSP frontiers and the interaction between IT and other organizational units in the planning process. The two case studies show that the incorporation of BSC provides synergies between different groups (such as Top Management, IT, and Exchange and Authorizing Area) in the development of key activities to strategically direct IT in both organizations. These

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

Using the BSC for Strategic Planning of IT in Brazilian Organizations

375

synergies were possible, in part, due to the need to consider the different perspectives from which IT strategies and actions can be described, including customers, financial, internal processes, and learning and growth. In this way, this work extends the contributions provided by contributes to Segars and Groverâ&#x20AC;&#x2122;s (2005) with respect to ITSP process definition and comprehending how it can provide satisfactory results. The alignment between BSC and ITSP offers a systematic approach that guides the definition of IT directions in alignment with business units and strategic priorities. The proposed method provides generic guidelines that organizations can use to incorporate balanced scorecard concepts in the ITSP. Then, they can create an environment that is more conductive to the necessary synergies between groups when developing IT strategies that are aligned with the business. However, the method itself is not a sufficient condition to overcome ITSP challenges. Skilled people, organizational culture, and other factors also exert significant influence and must be investigated in future works. This study has some natural limitations in terms of study focus. Two organizations were studied; however, these findings cannot be used to generalize the results and conclusions obtained from this study to any other specific organization. Moreover, this work is limited to ITSP development and does not cover implementation from developing strategies outside of the ITSP process. As one aim of this study is to establish an understanding of the applicability of BSC on ITSP, the following future studies are proposed. Future works can expand the context of the proposed method to other organizations and contexts, aiming to deepen the understanding of balanced scorecard applicability on ITSP. Additionally, future researchers can adopt a quantitative approach to define and test hypotheses about BSC application on ITSP. Finally, future works can evaluate the implementation results of IT strategic plans developed using balanced scorecard concepts. Such studies can be relevant to evaluate the contributions of BSC for approximating IT strategies, implementation, and planning activities.

Acknowledgements The authors thank the National Council of Technological and Scientific Development (CNPq â&#x20AC;&#x201C; Brazil), Coordination for the Improvement of Higher Level Personnel (Capes - Brazil), and the Foundation for Research Support of the State of Minas Gerais (FAPEMIG - Brazil) for the financial support.

A prior version of this paper was presented at the 9th International Conference on Information System and Technology Management (CONTECSI) in 2010 in SĂŁo Paulo, Brazil. The authors also thank the double-blind reviewers of this conference. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

376 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

REFERENCES Bechor, T., Neumann, S., Zviran, M., Chanan, G., & (2010). A contingency model for estimating success of strategic information systems Planning. Information & Management., 47(1), 17-29. Bermejo, P. H. S., Tonelli, A. O., Brito, M., J., & Zambalde, A. L. (2012).Implementation of information technology governance through IT strategic planning. African Journal of Business Management, 6: 11179-11189. Boar, B. H. (1994).Practical steps for aligning information technology with business strategies: how to achieve a competitive advantage. New York: John Wiley. Broady-Preston, J., & Hayward, T. E. (1998).An assessment of the relationship between marketing, information and strategy formulation in the retail banking sector. International Journal of Information Management, 18(4), 277-285. Cerpa, N., & Verner, J. M. (1998). Case study: The effect of IS maturity on information systems strategic planning. Information & Management, 34(2), 199-208. . Earl, M. J. (2003). Approaches to information systems planning. Experiences in strategic information systems planning. In R. D. Galliers & D. E. Leidner (Eds.), Strategic information management: challenges and strategies in managing information systems (Third ed.). Burlington: Elsevier. Gimbel, S. (1968).Exploring the Scientific Method: Cases and Questions. London: The University of Chicago Press. Grover, V., & Segars, A. H. (2005). An empirical evaluation of stages of strategic information systems planning: patterns of process design and effectiveness. Information & Management, 42(5), 761-779. Gutierrez, A., & Serrano, A. (2008). Assessing Strategic, Tactical, and Operational Alignment Factors for SMEs: Alignment Across the Organisation's Value Chain. International Journal of Value Chain Management, 2(1), 33-56. Heckman, R. (2003). Strategic Information Technology Planning and the Line Manager's Role. Information Systems Management 20(4), 16-21. Huang, C., & Hu, Q. (2007).Achieving IT-business strategic alignment via enterprisewide implementation of balanced scorecards. Information Systems Management 24(2), 173-184. ITGI, I. G. I. (2007).COBIT 4.1: control objectives, management guidelines, maturity models. Rolling Meadows: ITGI. Jung, C. F. (2004). Metodologia para pesquisa & desenvolvimento aplicada a novas tecnologias, produtos e processos. Rio de Janeiro: Axcel Books do Brasil. Kaplan, R. S., & Norton, D. P. (1997). A estratégia em ação: balanced scorecard (22th ed.). Rio de Janeiro: Elsevier.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

Using the BSC for Strategic Planning of IT in Brazilian Organizations

377

Kaplan, R. S., & Norton, D. P. (2008). A Execução Premium: A obtenção de vantagem competitiva através do vínculo da estratégia com as operações de negócio. Rio de Janeiro: Elsevier. Keyes, J. (2005). Implementing the IT Balanced Scorecard: Aligning IT with Corporate Strategy. Boca Raton: Auerbach Publications. Lederer, A. L., & Salmela, H. (1996).Toward a theory of strategic information systems planning.Journal of Strategic Information Systems, 5(3), 237-253. Lederer, A. L., & Sethi, V. (2003). The Information Systems Planning Process: Meeting the challenges of information systems planning. In R. D. Galliers & L. D. E. (Eds.), Strategic information management: challenges and strategies in managing information systems United Kingdom: Elsevier. Lee, G. G., & Bai, R. J. (2003). Organizational Mechanisms for successful IT/IS strategic planning in the digital era. Management Decision, 41(1), 32-42. Lee, G. G., & Pai, R. J. (2003).Effects of organizational context and inter-group behaviour on the success of strategic information systems planning: an empirical study. Behaviour and Information Technology, 22(4), 263-280. Littler, K., Aisthorpe, P., Hudson, R., & Keasey, K. (2000). A new approach to linking strategy formulation and strategy implementation: an example from the UK banking sector. International Journal of Information Management, 20, 411-428. Lutchen, M. D. (2004). Managing IT as a business: a survival guide for CEOs. Hoboken: John Wiley & Sons. Mentzas, G. (1997). Implementing is an strategy: a team approach. Long Range Planning, 30(1), 84-95. Min, S. K., Suh, E. H., & Kim, S. Y. (1999).An integrated approach toward strategic information systems planning. Journal of Strategic Information Systems, 8(4), 373-394. Newkirk, H. E., L., L. A., & Srinivasan, C. (2003). Strategic information systems planning: too little or too much. Journal of Strategic Information Systems, 12(3), 201228. Niven, P. (2002). Balanced Scorecard Step by Step: Maximizing Performance and Maximizing Results. New York: John Wiley & Sons. Pai, J. C. (2006). An empirical study of the relationship between knowledge sharing and IS/IT strategic planning (ISSP). . Management Decision, 44(1), 105-122. Salmela, H., & Spil, T. A. M. (2002).Dynamic and emergent information systems strategy formulation and implementation. International Journal of Information Management, 22, 441-460. Segars, A. H., & Grover, V. (1998). Strategic information systems planning success: an investigation of the construct and its measurement. MIS Quarterly, 22(2), 139-163. Simon, P. P. (2011). Design and implementation of the Balanced Scorecard at a university institute. Measuring Business Excellence, 15(3), 34 - 45.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

378 Tonelli, A. O., Bermejo, P. H. de S., Zambalde, A. L.

Van Grembergen, W., & De Haes, S. (2005). Measuring and Improving IT Governance Through the Balanced Scorecard. Information Systems Control Journal 2. Velcu, O. (2010). Strategic alignment of ERP implementation stages: An empirical investigation. Information & Management, 47(3), 158-166. Yin, R. (2005). Estudo de caso: planejamento e mĂŠtodos (Third ed.). SĂŁo Paulo: Artmed.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 361-378

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 379-396 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200009

CREDIT ANALYSIS USING DATA MINING: APPLICATION IN THE CASE OF A CREDIT UNION Marcos de Moraes Sousa Instituto Federal Goiano Campus Ceres, Goiás, Brasil Reginaldo Santana Figueiredo Universidade Federal de Goiás, Goiás, Brasil _____________________________________________________________________ ABSTRACT The search for efficiency in the cooperative credit sector has led cooperatives to adopt new technology and managerial knowhow. Among the tools that facilitate efficiency, data mining has stood out in recent years as a sophisticated methodology to search for knowledge that is “hidden” in organizations' databases. The process of granting credit is one of the central functions of a credit union; therefore, the use of instruments that support that process is desirable and may become a key factor in credit management. The steps undertaken by the present case study to perform the knowledge discovery process were data selection, data pre-processing and cleanup, data transformation, data mining, and the interpretation and evaluation of results. The results were evaluated through crossvalidation of ten sets, repeated in ten simulations. The goal of this study is to develop models to analyze the capacity of a credit union's members to settle their commitments, using a decision tree—C4.5 algorithm and an artificial neural network—multilayer perceptron algorithm. It is concluded that for the problem at hand, the models have statistically similar results and may aid in a cooperative's decision-making process. Keywords: Credit Unionism; Data Mining; Decision Tree; Artificial Neural Network.

_____________________________________________________________________________________________ Manuscript first received/Recebido em: 17/07/2012 Manuscript accepted/Aprovado em: 21/03/2014 Address for correspondence / Endereço para correspondência Marcos de Moraes Sousa, Professor in Administration, Federal Institute of Goiás; Ph.D. candidate, University of Brasília. Works on projects related to public administration, quantitative methods, and analysis of organizational performance. E-Mail: lceara@hotmail.com Reginaldo Santana Figueiredo, Post-doctoral fellow in Modeling and Simulation, Department of Industrial and Systems Engineering, Texas A&M University, IE-TAMU, USA (2002), Ph.D., Industry Economics, Federal University of Rio de Janeiro (UFRJ); Associate Professor, Federal University of Goiás—UFG. He was also a consultant for the Brazilian government on the analysis of production chains and for the Department of Recreation, Park and Tourism Science of Texas A&M University, in the analysis of tourism's socioeconomic impact, using modeling and simulation techniques. E-mail: santanrf@uol.com.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

380 Sousa, M. de M., Figueiredo, R. S.

1. INTRODUCTION This article describes the development of models to analyze the capacity of a credit union's associates to settle their commitments. Data mining techniques were used to develop the models. To construct the models, the actual database of cooperative borrowers from a SICOOB (Sistema Cooperativo Brasileiro—Brazilian Cooperative System) credit union was used. It must be stressed that such data are difficult to access and collect. A credit cooperative or union is a society and must be guided by a social purpose. However, it is also a financial institution and is regulated by norms imposed by the National Monetary Council of Brazil and the Brazilian Central Bank. Moreover, a credit union must aim to remain in the market permanently, which requires that resources be managed efficiently. Cooperative Movement may be classified according to two concepts: the Rochdale principle, which aims to transform society and reform man; and the theoretical principle, developed at the University of Münster (Germany), which uses the tools of business administration science and views the cooperative as a modern enterprise (Pinho, 2004). From the theoretical perspective, the Münster theory is the best developed. It is also known as the economic theory of cooperative cooperation, and its origins lie in the Institute for Cooperative Research at the University of Münster in Germany. In opposition to the Rochdale principle’s doctrinal assumptions, professors at the University of Münster, together with Latin American scholars, developed a “school” with methodological foundations that trace back to critical rationalism (Pinho, 2004). Pinho (1982, p. 75), following Bettcher, exposes the following concept of the cooperative, based on the axioms and assumptions of the Münster theory: “Cooperatives are groups of individuals that defend their individual economic interests by means of a company that they jointly maintain”1. In this context, Frantz (1985:56) adds that the cooperative may also be understood as embodying a “[...] competition strategy aimed to maximize the results of each producer's individual economic action [...]”. This study looks at credit unions from the perspective of the theoretical cooperative movement and, based on its assumptions and axioms, it finds that analyzing information for decision-making is a central condition. Tools and methodologies aimed at analyzing managerial information have greatly evolved in recent decades. Credit-union management is very complex because it must balance cooperative members' yearnings and needs while competing in the market. A credit union’s characteristics of being an association of members, while also being a company in the market, must remain balanced.

All direct quotations were translated by the authors.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

381

The number of cooperative members and unions has been slowly increasing. According to data from the Organization of Brazilian Cooperatives (Organização das Cooperativas Brasileiras—OCB) (2014), there are currently more than 1,047 single credit cooperatives and 4,529 points of service in Brazil. The SICOOB is the largest cooperative credit system in Brazil, with 529 single cooperatives and 1,949 cooperative service points (Portal do Cooperativismo de Crédito, 2014). The dynamic and competitive environment of Brazil's financial market, together with recent changes in credit offerings, demands a professional posture, which leads credit unions to adopt the use of new technologies and management knowhow. Oliveira (2001) indicates that the professionalization of cooperative members and unions is a relevant trend. The sector has developed quickly, adopting an integration strategy using central cooperatives; thus, it must be tuned in to the most efficient management tools. Credit analysis is one of the most important issues for financial institutions. Chaia (2003) stresses the importance of defining the type of analysis to be conducted and its coverage, and further warns of the dangers of copying and using other institutions' models, thus arriving at inadequate assessments. One of the foremost methods of credit assessment used by financial institutions is credit scoring. Chaia (2003, p. 23) defines this model as the use of statistical tools to identify the factors that determine the probability of a client going into default and notes that their main advantage is that “[...] grant-related decisions are made based on impersonal and standardized procedures, generating a higher degree of reliability”. Given the previously discussed concept of cooperative, the use of such objective methodologies—such as credit scoring—in granting credit is highly relevant. Decisions made solely by evaluating subjective assessments are avoided. Koh, Tan, and Goh (2006) tie the progress of credit scoring to increased competitiveness, advances in computational technology, and the exponential growth of large databases. Mester (1997) notes that model exactness, current data, and model evaluation and readjustment are some critical factors in credit scoring and that flaws in these factors limit the use of the model. Given that lending is one of credit unions' central functions, the analysis of that function is fundamental to protecting the cooperative's collective assets. Obtaining tools that classify and help predict the behavior of future loans is fundamental to credit management, helping to reduce process subjectivity, allowing more efficient resource allocation, and resulting in quicker responses to proposals. Studies on credit analysis using data mining increase the models' precision and have been conducted by several authors in recent years (Abellán & Mantas, 2014; Akkoç, 2012; Bhattacharyya, Jha, Tharakunnel, & Christopher, 2011; Chang & Yeh, 2012; Chen & Huang, 2011; Crone & Finlay, 2012; Cubiles-De-La-Vega, BlancoOliver, Pino-Mejías, & Lara-Rubio, 2013; García, Marqués, & Sánchez, 2012; Han, Han, & Zhao, 2013; Koh et al., 2006; Kruppa, Schwarz, Arminger, & Ziegler, 2013; Lai, Yu, Wang, & Zhou, 2006; Lemos, Steiner, & Nievola, 2005; Majeske & Lauer, 2013; Marqués, García, & Sánchez, 2012; Nie, Rowe, Zhang, Tian, & Shi, 2011;

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

382 Sousa, M. de M., Figueiredo, R. S.

Oreski & Oreski, 2014; Saberi et al., 2013; Wang, Ma, Huang, & Xu, 2012; Xiong, Wang, Mayers, & Monga, 2013; Yap, Ong, & Husain, 2011; Zhong, Miao, Shen, & Feng, 2014; Zhou, Jiang, Shi, & Tian, 2011; Zhu, Li, Wu, Wang, & Liang, 2013). Despite growing interest, there is still little application of this tool in cooperatives. Khatchatourian and Treter (2010) apply fuzzy logic to their analysis of the financial performance of production cooperatives in the Brazilian state of Rio Grande do Sul. Zhu, Li, Wu, Wang, and Liang (2013) use a support vector machine in their credit analysis of a credit union in Barbados. Currently, there are several data mining techniques available. Therefore, the intention here is to examine which data mining methodology provides the best creditanalysis results for credit unions. To this end, this study’s objective is to determine whether a data mining model can perform well for classifying and predicting credit unions' credit management. 2. THEORETICAL FRAMEWORK The term knowledge discovery in databases (KDD) was first used in 1989 to stress that knowledge is the final product of the discovery process in databases (Fayyad, Piatetsky-Shapiro, & Smyth, 1996). Until 1995, the terms KDD and data mining were understood by many authors as synonymous (Lemos et al., 2005). Fayyad et al. (1996) defines and distinguishes KDD and data mining as follows: the former refers to the general process of discovering useful knowledge from data, whereas the latter refers to the specific application of algorithms to extract patterns and models from data. In the view of these authors, data mining thus is a step in the KDD process that consists of applying algorithms in the production of a particular set of patterns and models. Fayyad et al. (1996) refer to patterns as model components. This study used the concept of model defined by Pidd (1998, p. 23): “A model is an external and explicit representation of part of a reality, observed by a person who wishes to use that model to understand, alter, manage, and control part of that reality”. Goldschmidt and Passos (2003, p. 6) split KDD activities into three groups: (i) technological development, a group that incorporates “[...] the initiatives for conception, betterment and development of algorithms, support tools and technologies [...]” into the KDD process; (ii) KDD execution, a group composed of activities related to using algorithms, tools, and technologies developed in the search of knowledge; (iii) application of results, as a group that consists of the models developed in the KDD execution, in a such way that “[...] the activities turn toward the application of results in the context in which the KDD process was conducted”. Hybrid or composite techniques and models are usually compared. The following models of credit and risk analysis stand out: (a) logistic regression (Akkoç, 2012; Bhattacharyya et al., 2011; Cubiles-De-La-Vega et al., 2013; Han et al., 2013; Ju & Sohn, 2014; Koh et al., 2006; Kruppa et al., 2013; Nie et al., 2011; Wang et al., 2012; Yap et al., 2011); (b) decision trees (Abellán & Mantas, 2014; Bhattacharyya et al., 2011; Chen & Huang, 2011; Crone & Finlay, 2012; Koh et al., 2006; Kruppa et al., JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

383

2013; Lemos et al., 2005; Nie et al., 2011; Wang et al., 2012; Yap et al., 2011); (c) neural networks (Akkoç, 2012; Chen & Huang, 2011; Koh et al., 2006; Lai et al., 2006; Nie et al., 2011; Oreski & Oreski, 2014; Saberi et al., 2013; Wang et al., 2012); (d) support vector machines (Bhattacharyya et al., 2011; Nie et al., 2011; Xiong et al., 2013; Zhong et al., 2014; Zhu et al., 2013); and (e) ensemble methods (Abellán & Mantas, 2014; García et al., 2012; Marqués et al., 2012; Nie et al., 2011; Wang et al., 2012). The studies' real application contexts were obtained from organizations in Canada (Xiong et al., 2013), Germany (Han et al., 2013; Koh et al., 2006), Croatia (Oreski & Oreski, 2014), Peru (Cubiles-De-La-Vega et al., 2013), China (Nie et al., 2011), Turkey (Akkoç, 2012), and Barbados (Zhu et al., 2013). Credit analysis using data mining is still rare in Brazil: Lemos et al. (2005) find the same methodology being employed for bank credit analysis, using a branch office of Banco do Brasil as their locus. Decision trees are one of the most prominent and popular data mining methods (Wang et al., 2012). According to Lemos et al. (2005, p. 229), a decision tree is the only method that provides results in a hierarchical manner; i.e., “[...] the most relevant attribute is placed in the first node of the tree, and less relevant attributes are placed in subsequent nodes”. A decision tree is therefore a structure that is used to split a large amount of data into successive, smaller sets by applying a sequence of decision rules (Berry & Linoff, 2004). The construction of decision trees is especially attractive in the KDD context, which according to Gehrke (2003) is attributable to the following reasons: intuitive and easy-to-understand results; non-parametric properties that are therefore applicable to exploratory treatments; relatively fast construction when compared to other methods; and accuracy that can be compared to the accuracy of other models. Decision trees are commonly converted into decision rules. A decision tree may be observed as: [...] a graph in which each non-leaf node represents a predicate (condition) involving an attribute and a set of values. The leaf nodes correspond to the attribution of a value or set of values to a problem attribute (Goldschmidt & Passos, 2005, p. 57). In light of this observation, paths in the tree correspond to rules of the type, “IF <conditions> THEN <conclusion>”. Several algorithms have been developed based on the induction of decision trees, among which the following stand out: C4.5, CART (classification and regression trees), QUEST (quick, unbiased, efficient statistical trees) and CHAID (chi-square automatic interaction detectors). An artificial neural network (ANN) is a mathematical model based on the brain structure, ordered into layers and connections. The origin of ANNs dates back to 1943, but it was in the 1980s that greater interest in the method appeared, its development

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

384 Sousa, M. de M., Figueiredo, R. S.

fostered mainly by advances in information technology (Braga, Carvalho, & Ludermir, 2000). In Goldschmidt and Passos's (2005, p. 175) view, ANNs can be observed as “[...] mathematical models inspired by the working principles of biological neurons and the brain's structure”. Such models, according to those authors, allow the simulation of human abilities such as learning, generalizing, associating, and abstracting. Braga et al. (2000, p. 1) define ANNs as “distributed parallel systems composed of simple processing units (nodes) that compute some given (usually non-linear) mathematical function, [...] arranged into one or more layer and interconnected by a large number of connections [...]” An ANN's structure therefore consists of neuron layers and weighted connections. As shown in Figure 1, neurons are represented by nodes and weighted connections are represented by arrows.

Figure 1 – ANN architecture Source: Goldschmidt & Passos (2005). Typically, there are three stages of ANN processing: the input layer, in which the data are received; the internal layer, usually called the hidden layer, which is responsible for processing the data and may consist of more than one actual layer; and the output layer, which provides the result (Larose, 2005). The first step when applying ANNs is the network's learning phase, in which parameters are adjusted. This learning may be of two types: supervised or unsupervised. The first type occurs when output (or target) variables' values are provided, the second type occurs in the absence of those values. Braga et al. (2000, p. 227) mention as positive points, which arouse interest in the method, the ability to learn and later to generalize the possibility of mapping

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

385

multivariable functions, self-organization, the process of time-series, the possibility of using a large number of input variables, and the possibility of using samples. Because the model is considered non-parametric, these authors further stress, “[...] there is no great need to understand the process itself”. However, these authors also consider this last aspect to constitute the primary criticism of the model; i.e., the model's inability to clarify how its results are generated. Due to this peculiarity, ANNs are also called “black boxes”. 3.

METHODOLOGY

In this research, we opted to use case studies, which according to Yin (2010), are adequate to study contemporary events in a real-life context when controlling variables becomes more difficult for the researcher. This case study is unique and contemplates one unit of analysis, which involves one Credit Union of the SICOOB system—structured to contemplate investigation into the union from the perspective of theoretical cooperative movement. The cooperative's database was used to evaluate the credit analysis system's performance. This database comprises the historical data of natural persons' analyses from 2003 to 2007. Due to a change in the information system, it was not possible to gather data from before this period. Data referring to credit analysis is highly confidential and strategic, due to banking secrecy and the risk of competitors acquiring the data. This makes such data very difficult for third parties to obtain. The choice to study a cooperative was therefore due to its willingness to provide the data. The cooperative currently uses the SisBr application of the SICOOB system as a credit analysis tool. This application contains the information used by management and the board of directors to make decisions about whether to grant credit. The study followed the steps suggested by Fayyad, Piatetsky-Shapiro, and Smyth (1996) for the knowledge discovery process: data selection, data pre-processing and cleanup, data transformation, data mining, data interpretation, and the evaluation of results. To reach its goals, this study was based on the activities involved in conducting KDD, as discussed by Goldschmidt and Passos (2005). Among the available data mining techniques, neural networks and decision trees were used, both of which are common in empirical studies. Data collection and selection corresponded to the process of capturing, organizing, and selecting the data made available for the modeling and data-mining phases, and thus required accurate examination. Dasu and Johnson (2003) note the following factors that are helpful in analyzing variables: previous experience, knowledge, quantity of results, and quality of results.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

386 Sousa, M. de M., Figueiredo, R. S.

3. RESULTS This section describes the simulation results of the techniques investigated in this articleâ&#x20AC;&#x201D;namely, the decision tree and ANN techniquesâ&#x20AC;&#x201D;along with the statistical tests comparing them. No missing values are found in the database. Chart 01 shows the structure of the constructed database, together with its variables and their possible values. Chart 01 - Database structure CODE 01

VARIABLE Member code

Gender

Age

Level of education

05 06

City Birthplace

Place of residence

Primary work/activity

Marital status

Capital

Relationship

Transaction conduct

Years of experience in activity/work

Records check

Record information at the cooperative

Purpose of the operation

VALUE Numerical key, unique for each member 1= male 2= female Numerical value 1= graduate 2= complete tertiary 3= incomplete tertiary 4= complete secondary 5= incomplete secondary 6= complete primary 7= incomplete primary Name of city of residence Name of city of birth 1= urban area 2= rural area Name of work/activity 1= Married 2= Single 3= Widowed 4= Legally divorced 5= Other Numerical value 1= with the cooperative for more than 3 years 2= with the cooperative for 1 to 3 years 3= with the cooperative for less than 1 year 1= normal 2= occasional delays 3= constant delays/renegotiations 1= more than 5 years 2= from 3 to 5 years 3= less than 3 years 1= no restrictions 2= justified, irrelevant restrictions 3= relevant or unjustified, irrelevant restrictions 1= up-to-date and reliable record 2= up-to-date and unreliable record 3= information is not up-to-date or is missing 1= support and investment 2= financing of assets 3= personal credit/automatic loan 4= debt renewal/composition

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

387

Chart 01 - Database structure (Cont.) CODE

VARIABLE

Operation guarantees

Liquidity guarantee

Frequency with which the member performs (active) transactions

Operation value

Level of commitment—installments on member's net income

Personal net assets minus total liability

Total liability in relation to annual net income

Total liability in relation to paid-in capital

Profile of member's economical activity

Risk attributed by the cooperative

Result—July 2007

Result—August 2007

Result—September 2007

Result—October 2007

Result—November 2007

Result—December 2007

VALUE 1= mortgage—social capital 2= chattel mortgage/warrants 3= pledge/collateral 4= personal 1= high liquidity guarantee (sale in less than 6 months) 2= moderate liquidity guarantee (sale in 6 to 12 months) 3= personal or low liquidity guarantee (sale in more than 12 months) 1= never 2= frequently 3= permanently 1= up to 1% of adjusted net worth (ANW) 2= from 1.01% to 2% of ANW 3= from 2.01% to 3% of ANW 4= more than 3% of ANW 1= up to 20% of average net income 2= from 20%-30% of average net income 3= more than 30% of net income 1= more than 4 times 2= from 2 to 4 times 3= no personal equity or less than 2 times 1= less than 2 times 2= from 2 to 4 times 3= more than 4 times 1= less than 4 times 2= from 4 to 8 times 3= from 8 to 12 times 4= more than 12 times 1= excellent 2= good 3= regular 4= poor 1= AA 2= A 3= B 4= C 5= D 6= E 7= F 8= G 9= H 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

388 Sousa, M. de M., Figueiredo, R. S.

Chart 01 - Database structure (Cont.) CODE 33

VARIABLE Result—January 2008

Result—February 2008

Result—March 2008

Result—April 2008

Result—May 2008

Result—June 2008

Aggregate result

VALUE 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default 1= good standing 2= default

Source: Study data, 2010

The output vector is given by variables 27 to 39, corresponding to the period from July 2007 through June 2008. Variables 02 and 10 are not part of the credit analysis conducted by the cooperative and were added to broaden the analysis. These variables were gathered from cooperative members' records and represent the data made available by the cooperative. Variables 11 through 26 are currently used for credit analysis and represent the cooperative member's borrowing history. Codes 27 through 39 represent the output variable adopted in the study and depict the period from July 2007 through June 2008. These are the data that the cooperative provided. Older data were not available. The number of variables used in the analysis was consistent with other studies. For example, Koh et al. (2006) used 20 variables, and Lemos et al. (2005) used 24. The variable “cooperative member code” was discarded because it was useful only to identify individuals while collecting data. The variable “attributed risk” was used only in the data pre-processing and cleanup stages; it was not used in the transformation and modeling stages because it refers to the output of the model used by the cooperative, and therefore it represents the result of the model currently being used. The variable “aggregate results” represents the result of the period of analysis (July 2007 through June 2008) and according to the cooperative's business rules and the study's goals, is the model's target output variable. Data transformation aims to help carry out the data mining techniques. As recommended by Goldschmidt and Passos (2005), data were grouped into a single twodimensional table. The data were collected from two sources in the cooperative: first, from credit assessment; and second, from records gathered manually, record by record. In this survey, historical data from 211 individual members were used, out of which 22 were in default and 189 were in good standing. This data represents all of the cooperative’s member-borrowers. Given the difference in numbers between default and good

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

389

standing, there may be bias and overfitting problems (Chawla, 2005; Horta, Borges, Carvalho, & Alves, 2011). To solve this problem, a technique called SMOTE (synthetic minority oversampling technique) (Chawla, 2005) was used to insert observations of cooperative members in default. This algorithm is one of the most used in the literature (Horta et al., 2011). Thus, 110 observations of the minority class—i.e., members in default—were created, and the sample totaled 321 observations, out of which 132 were in default and 189 were in good standing. Next, the database was randomized to avoid a concentration of the same values into a given data set while cross-validating, which would have led to overfitting. For the computational implementation of the decision tree and neural network techniques, the database described in this study was used, taking into consideration, for each cooperative member, the previously described variables. For example, a model generated by the decision tree technique was selected for transformation into decision rules during the post-processing phase. The public domain computational tool WEKA (Waikato environment for knowledge analysis) was chosen to perform this task. Goldschmidt and Passos (2005, p. 50) argue that for a more reliable evaluation of the knowledge model, “[...] the data used in constructing the model should not be the same as used in this model's evaluation”. Those authors further state that there should be at least two divisions: training and testing. The first division comprises the data used in constructing the model; the second division comprises the data for evaluation. Splitting the data set served to simplify, summarize and reduce the database's size and variability, resulting in the selection of more sophisticated and accurate models (Dasu & Johnson, 2003). In this study, to increase assessment neutrality, K-fold cross-validation was used for both the decision tree and the ANN. According to Goldschmidt and Passos (2005, p. 51), in this method, the database is randomly split with N elements into K separate subsets: “each of the K subsets is used as a testing set, and the remaining (K-1) subsets are combined into a training set. The process is repeated K times, so that K models are generated and evaluated [...]”. The data were split into ten sets and repeated across ten simulations as proposed by Witten and Frank (2005). Cross-validation has been found in several studies on credit analysis (Akkoç, 2012; Chang & Yeh, 2012; Han et al., 2013). For this study, multi-layer ANNs, i.e., multilayer perceptron (MLP), were used with the back propagation learning algorithm. The number of neurons in the input layer was 66, plus two in the intermediate and two in the output layers. For all tests, a learning rate of 0.01 was used, given that this rate improved classification as observed in the simulations and was also used by Lemos et al., (2005). The momentum rate was not used as in Lemos et al. (2005); moreover, adding this rate did not improve classification performance. Supervised learning was used in this ANN. Ferreira (2005, p. 37) describes this type of learning as follows: “[...] the network is trained by supplying it with input values and the respective output values [...].

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

390 Sousa, M. de M., Figueiredo, R. S.

For a comparative analysis of the models, the total percentage of correctly predicted values was used as the parameter in the two-tailed corrected resample t-test, at a significance level of 0.05 (or 5%), with nine degrees of freedom, as proposed by Witten and Frank (2005) in equation 1, presented below:

(1) where

difference between mean numbers of correctly predicted values between the models. number of sets times number of repetitions. number of samples for testing. number of samples for training. variance of the difference between means.

3.1 Decision Tree In this study, the J4.8 tool was chosen for use. It is the WEKA implementation of the C4.5 decision tree algorithm. According to Goldschmidt and Passos (2005), this tree is broadly used and accepted. A model generated by the decision tree technique was taken to exemplify the rules and confusion matrix. The model below generated 41 leaves, i.e., sets of decision rules of the if-then type. Some rules of the first set are shown below:  If liquidity guarantee = high guarantee liquidity (sale in less than 6 months), then default;  If liquidity guarantee = moderate guarantee liquidity (sale in 6 to 12 months) and level of commitment = up to 20% of the average net income, then good standing;  If liquidity guarantee = moderate guarantee liquidity (sale in 6 to 12 months) and level of commitment = from 20% to 30% of the average net income, then good standing.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

391

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

Chart 02 shows the confusion matrix generated by the testing set of the tree being evaluated. This matrix shows the instances classified as predicted and actual to assess the models' hit and miss types. The main diagonal contains the correctly classified values. The values are given as absolute numbers.

Chart 02 - Confusion matrix of a model developed using the decision tree method Predicted Default Actual

Default Good standing

Good standing 121

181

Source: Study data, 2010.

In this example, the model based on the C4.5 decision tree algorithm correctly classified 302 records, which corresponds to accuracy rate of 94.08%, and incorrectly classified 19 observations, or 5.92%. 3. 2. Neural Network The ANN in this study, as previously discussed in the theoretical framework, contained three layers: input, intermediate, and output. The network used supervised learning because the model's output values were supplied. As in the case of the decision tree, a model was used to exemplify the results generated by the WEKA package. Chart 03 shows the confusion matrix for the model obtained.

Chart 03 - Confusion matrix for a model developed using the ANN method Predicted Default Actual

Good standing

Default

118

Good standing

176

Source: Study data, 2010.

This ANN-based model, constructed using the MLP algorithm, classified 294 records correctly and 27 records incorrectly, corresponding to 91.59% and 8.41% accuracy, respectively.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

392 Sousa, M. de M., Figueiredo, R. S.

3. 3. Model Evaluation This section comparatively evaluates the two models developed in this study: ANN (MLP algorithm) and decision tree (C4.5 algorithm). To conduct this evaluation, the total percentage of correctly predicted values was used. Table 01 shows the percentage result of correctly predicted values and the respective standard deviation of the simulations carried out using the studied models. Table 01â&#x20AC;&#x201D;Comparative evaluation Algorithm

C4.5

MLP

Correct percentage

97.07%

95.58%

Standard deviation

2.76

3.47

Source: Study data, 2010.

The simulations performed using WEKA's experimenter tool indicate that the decision tree technique implementing the C4.5 algorithm is statistically similar, according to a two-tailed test, to the ANN with the MLP algorithm at the 0.05 significance level. Figure 02 shows WEKA's output for the simulations of the previously described problem.

Figure 02: Output for the simulations run using WEKA's experimenter tool Source: Study data, 2010.

The decision tree's performance for the current problem is better than the one obtained by Yap et al. (2011), in which the error rate is 28.1%. The study by Lemos et al. (2005), despite not performing statistical tests, obtains a higher hit rate with the neural network than with the decision tree. Although statistically similar, decision trees are considered to be easy to use (Lemos et al., 2005). Our results indicate that the classification models based on data mining developed herein may be useful to the cooperative in its assessments, thus improving

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

393

performance, as previously found in the analysis of a microcredit organization (Cubiles-De-La-Vega et al., 2013) and a credit union (Zhu et al., 2013). 4. CONCLUSIONS The goal of this study is to develop and evaluate data mining models to classify and predict the behavior of cooperative members' behavior in honoring their obligations. The decision tree and ANN, both data mining techniques, were used to develop the model. The process of data preparation and modeling followed the steps suggested in the literature: data selection, data pre-processing and cleanup, data transformation, data mining, interpretation, and validation of the results. The data were divided into training and testing sets. Although the decision tree's accuracy in the simulations is 97.07%, compared to 95.58% with the ANN, the decision-tree-based C4.5 algorithm obtains a result that is statistically similar to that of the model that was based on the MLP artificial neural network. The knowledge discovery process and the use of models based on data mining developed here may provide the cooperative with practical advantages. Understanding the variables and their relationships may help in better classifying and predicting cooperative members' behavior. In-depth assessment of the variables may further help in including variables that might be important and excluding others that turn out not to be relevant, with the advantage of providing more succinct and precise credit management models, reducing execution time and improving decision accuracy. The analysis of discrepant or outlier cases may be relevant to creating a new classification or, conversely, finding undesirable patterns. This study is limited by the following issues: a lack of other cooperatives' databases for comparison, evaluation, and validation of the model; limitations of the information system used by the cooperative, which precluded collecting input variable data from before 2003 and provided data referring to the output value only for the past six months; and the lack of integration between some database modules and electronic spreadsheets. The following proposals are left for future studies: using different databases to validate the credit analysis model; using other data mining techniques; using hybrid models, combining different techniques to improve classification and predictive performance; investment analysis, evaluating the type of error and the financial impact that the model has on the cooperative's profitability; and evaluation of discrepant cases, particularly those cases involving the variable "capital", to check for the existence of new patterns and classifications.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

394 Sousa, M. de M., Figueiredo, R. S.

REFERENCES Abellán, J., & Mantas, C. J. (2014). Improving experimental studies about ensembles of classifiers for bankruptcy prediction and credit scoring. Expert Systems with Applications, 41(8), 3825–3830. Akkoç, S. (2012). An empirical comparison of conventional techniques, neural networks and the three stage hybrid Adaptive Neuro Fuzzy Inference System (ANFIS) model for credit scoring analysis: The case of Turkish credit card data. European Journal of Operational Research, 222(1), 168–178. Berry, M. J. A., & Linoff, G. (2004). Data mining techniques: For marketing, sales and customer relationship management (2nd ed.). Indianapolis: wiley Publishing. Bhattacharyya, S., Jha, S., Tharakunnel, K., & Christopher, J. (2011). Data mining for credit card fraud: A comparative study. Decision Support Systems, 50(3), 602–613. Braga, A. de P., Carvalho, A. P. de L. F., & Ludermir, T. B. (2000). Redes neurais artificiais: Teoria e aplicações. Rio de Janeiro: LTC. Chaia, A. J. (2003). Modelos de gestão de risco de crédito e sua aplicabilidade ao mercado brasileiro. Dissertação de Mestrado. FEA/USP. Chang, S.-Y., & Yeh, T.-Y. (2012). An artificial immune classifier for credit scoring analysis. Applied Soft Computing, 12(2), 611–618. Chawla, N. V. (2005). Data mining for imbalanced datasets: An overview. In Data mining and knowledge discovery handbook (pp. 853–867). New Jersey: Springer. Chen, S. C., & Huang, M. Y. (2011). Constructing credit auditing and control & management model with data mining technique. Expert Systems with Applications, 38(5359-5365). Crone, S. F., & Finlay, S. (2012). Instance sampling in credit scoring: An empirical study of sample size and balancing. International Journal of Forecasting, 28(1), 224– 238. Cubiles-De-La-Vega, M.-D., Blanco-Oliver, A., Pino-Mejías, R., & Lara-Rubio, J. (2013). Improving the management of microfinance institutions by using credit scoring models based on Statistical Learning techniques. Expert Systems with Applications, 40(17), 6910–6917. Dasu, T., & Johnson, T. (2003). Exploratory data mining and data cleaning. New Jersey: John Wiley & Sons. Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). From data mining to knowledge discovery in databases. A I Magazine, 17(3), 37–54. Ferreira, J. B. (2005). Mineração de dados na retenção de clientes em telefonia celular. Dissertação de Mestrado. PUC-RIO. García, V., Marqués, A. I., & Sánchez, J. S. (2012). On the use of data filtering techniques for credit risk prediction with instance-based models. Expert Systems with Applications, 39(18), 13267–13276. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Credit Analysis Using Data Mining: Application In The Case of a Credit Union

395

Gehrke, J. (2003). Decision tree. In The handbook of data mining (pp. 3–23). New Jersey: Lawrence Erlbaum Associates. Goldschmidt, R., & Passos, E. (2005). Data mining: Um guia prático. Rio de Janeiro: Elsevier. Han, L., Han, L., & Zhao, H. (2013). Orthogonal support vector machine for credit scoring. Engineering Applications of Artificial Intelligence, 26(2), 848–862. Horta, R. A. M., Borges, C. C. H., Carvalho, F. A. A., & Alves, F. J. S. (2011). Previsão de insolvência: Uma estratégia para balanceamento da base de dados utilizando variáveis contábeis de empresas brasileiras. Sociedade, Contabilidade E Gestão, 6(2), 21–36. Ju, Y. H., & Sohn, S. Y. (2014). Updating a credit-scoring model based on new attributes without realization of actual data. European Journal of Operational Research, 234(1), 119–126. Khatchatourian, O., & Treter, J. (2010). APLICAÇÃO DA LÓGICA FUZZY PARA AVALIAÇÃO ECONÔMICO-FINANCEIRA DE COOPERATIVAS DE PRODUÇÃO. Revista de Gestão Da Tecnologia E Sistemas de Informação, 7(1), 141– 162. Koh, H. C., Tan, W. C., & Goh, C. P. (2006). A two-step method to construct credit scoring models with data mining techniques. International Journal of Business and Information, 1(1), 96–118. Kruppa, J., Schwarz, A., Arminger, G., & Ziegler, A. (2013). Consumer credit risk: Individual probability estimates using machine learning. Expert Systems with Applications, 40(13), 5125–5131. Lai, K. K., Yu, L., Wang, S., & Zhou, L. (2006). Credit risk analysis using a reliabilitybased neural network ensemble model. In Artificial Neural Networks-ICANN 2006 (pp. 682–690). Springer Berlin Heidelberg. Larose, T. D. (2005). Discovering knowledge in data: An introduction to data mining. New Jersey: John Wiley & Sons. Lemos, E. P., Steiner, M. T. A., & Nievola, J. C. (2005). Análise de crédito bancário por meio de redes neurais e árvore de decisao: Uma aplicação simples de data mining. Revista de Administração Da Universidade de São Paulo, 40(3), 225–234. Majeske, K. D., & Lauer, T. W. (2013). The bank loan approval decision from multiple perspectives. Expert Systems with Applications, 40(5), 1591–1598. Marqués, A. I., García, V., & Sánchez, J. S. (2012). Two-level classifier ensembles for credit risk assessment. Expert Systems with Applications, 39(12), 10916–10922. Mester, L. J. (1997). What’s the point of credit scoring? Business Review, 3, 3–16. Nie, G., Rowe, W., Zhang, L., Tian, Y., & Shi, Y. (2011). Credit card churn forecasting by logistic regression and decision tree. Expert Systems with Applications, 38(12), 15273–15285.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

396 Sousa, M. de M., Figueiredo, R. S.

OCB. (2014). Organização das Cooperativas Brasileiras. Números. Retrieved February 20, 2014, from http://www.ocb.org.br/site/ramos/credito_numeros.asp Oliveira, D. P. R. (2001). Manual de gestão de cooperativas: Uma abordagem prática. São Paulo: Atlas. Oreski, S., & Oreski, G. (2014). Genetic algorithm-based heuristic for feature selection in credit risk assessment. Expert Systems with Applications, 41(4), 2052–2064. Pidd, M. (1998). Modelagem empresarial: Ferramentas para tomada de decisão. São Paulo: Atlas. Pinho, D. B. (1982). O pensamento cooperativo e o cooperativismo brasileiro. CNPq/BNCC. Pinho, D. B. (2004). O cooperativismo no Brasil: Da vertente pioneira à vertente solidaria. São Paulo: Saraiva. Portal do Cooperativismo de Crédito. (2014). Dados consolidados dos sistemas cooperativos. Retrieved February 20, 2014, from http://cooperativismodecredito.coop.br/cenario-brasileiro/dados-consolidados-dossistemas-cooperativos/ Saberi, M., Mirtalaie, M. S., Hussain, F. K., Azadeh, A., Hussain, O. K., & Ashjari, B. (2013). A granular computing-based approach to credit scoring modeling. Neurocomputing, 122(25), 100–115. Wang, G., Ma, J., Huang, L., & Xu, K. (2012). Two credit scoring models based on dual strategy ensemble trees. Knowledge-Based Systems, 26, 61–68. Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques (2nd ed.). San Francisco: Elsevier. Xiong, T., Wang, S., Mayers, A., & Monga, E. (2013). Personal bankruptcy prediction by mining credit card data. Expert Systems with Applications, 40(2), 665–676. Yap, B. W., Ong, S. H., & Husain, N. H. M. (2011). Using data mining to improve assessment of credit worthiness. Expert Systems with Applications, 38(10), 13274– 13283. Yin, Robert, K. (2010). Estudo de caso: planejamento e métodos (4th ed.). Porto Alegre: Bookman. Zhong, H., Miao, C., Shen, Z., & Feng, Y. (2014). Comparing the learning effectiveness of BP, ELM, I-ELM, and SVM for corporate credit ratings. Neurocomputing, 128(27), 285–295. Zhou, X., Jiang, W., Shi, Y., & Tian, Y. (2011). Credit risk evaluation with kernelbased affine subspace nearest points learning method. Expert Systems with Applications, 38(4), 4272–4279. Zhu, X., Li, J., Wu, D., Wang, H., & Liang, C. (2013). Balancing accuracy, complexity and interpretability in consumer credit decision making: A C-TOPSIS classification approach. Knowledge-Based Systems, 52, 258–267.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

ANÁLISE DE CRÉDITO POR MEIO DE MINERAÇÃO DE DADOS: APLICAÇÃO EM COOPERATIVA DE CRÉDITO Marcos de Moraes Sousa Instituto Federal Goiano Campus Ceres, Goiás, Brasil Reginaldo Santana Figueiredo Universidade Federal de Goiás, Goiás, Brasil _____________________________________________________________________ RESUMO A busca por eficiência no setor cooperativista de crédito tem levado as cooperativas a adotarem novas tecnologias e novos conhecimentos gerenciais. Dentre tais ferramentas, a Mineração de Dados tem-se destacado nos últimos anos como uma metodologia sofisticada na busca de conhecimento “escondido” nas bases de dados das organizações. Entende-se que o processo de concessão de crédito é uma operação central de uma cooperativa de crédito, assim, o uso de instrumentos que auxiliem são desejados e podem tornar-se fator-chave na gestão do crédito. Os passos utilizados na execução do processo de Descoberta de Conhecimento do presente estudo de caso foram: seleção dos dados; pré-processamento e limpeza dos dados; transformação dos dados; Mineração de Dados; interpretação e avaliação dos resultados. Os resultados foram avaliados por meio de validação cruzada em dez conjuntos e repetidos em dez simulações. Este estudo propôsse a desenvolver modelos para analisar a capacidade dos cooperados de uma cooperativa de crédito de saldar seus compromissos, utilizando Árvore de Decisão – algoritmo C4.5 e Rede Neural Artificial – algoritmo Multilayer Perceptrom. Conclui-se que, para o problema proposto, os modelos tiveram resultados estatisticamente semelhantes e que podem auxiliar no processo decisório da cooperativa. Palavras-chave: Cooperativismo de Crédito; Mineração de Dados; Árvore de

Decisão; Rede Neural Artificial. _____________________________________________________________________________________________ Manuscript first received/Recebido em: 17/07/2012 Manuscript accepted/Aprovado em: 21/03/2014 Address for correspondence / Endereço para correspondência

Marcos de Moraes Sousa, Professor in Administration, Federal Institute of Goiás; Ph.D. candidate, University of Brasília. Works on projects related to public administration, quantitative methods, and analysis of organizational performance. E-Mail: lceara@hotmail.com Reginaldo Santana Figueiredo, Post-doctoral fellow in Modeling and Simulation, Department of Industrial and Systems Engineering, Texas A&M University, IE-TAMU, USA (2002), Ph.D., Industry Economics, Federal University of Rio de Janeiro (UFRJ); Associate Professor, Federal University of Goiás—UFG. He was also a consultant for the Brazilian government on the analysis of production chains and for the Department of Recreation, Park and Tourism Science of Texas A&M University, in the analysis of tourism's socioeconomic impact, using modeling and simulation techniques. E-mail: santanrf@uol.com.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

380 Sousa, M. de M., Figueiredo, R. S.

1. INTRODUÇÃO O presente artigo trata do desenvolvimento de modelos para analisar a capacidade dos associados de uma cooperativa de crédito de saldar os seus compromissos. Para tal, foram utilizadas técnicas de Mineração de Dados (Data Mining). Para a construção do modelo foi utilizada a base de dados real de cooperados tomadores de crédito de uma cooperativa de crédito do sistema SICOOB (Sistema Cooperativo Brasileiro). Ressalta-se que são dados de difícil acesso e coleta. O cooperativismo de crédito é uma sociedade de pessoas e deve ser norteado por uma finalidade social. Entretanto, é também, uma instituição financeira e é regulamentada pelas normas impostas pelo Conselho Monetário Nacional e pelo Banco Central e, ademais, deve também ter o objetivo de permanência no mercado, o que impõe uma gestão eficiente dos recursos. O cooperativismo pode ser classificado em duas vertentes: a doutrinária rochdaleana, que pretendia transformar a sociedade e reformar o homem; e a teórica, desenvolvida na Universidade de Münster (Alemanha), utilizando o instrumental da ciência da administração de empresa, vislumbrando a cooperativa como uma empresa moderna (Pinho, 2004). Na perspectiva teórica, a Teoria de Münster é a que mais se desenvolveu, também conhecida por “Teoria Econômica da Cooperação Cooperativa”, com origem no Instituto de Cooperativismo da Universidade de Münster, na Alemanha. Professores desta universidade, conjuntamente com pesquisadores latino-americanos, em oposição aos pressupostos doutrinários rochdaleanos, desenvolveram esta “Escola”, cuja fundamentação metodológica advém do racionalismo crítico (Pinho, 2004). Pinho (1982, p. 75) expõe, segundo as ideias de Boettcher, que o seguinte conceito de cooperativa baseado nos axiomas e pressupostos da Teoria de Münster: “as cooperativas são agrupamentos de indivíduos que defendem seus interesses econômicos individuais por meio de uma empresa que eles mantêm conjuntamente”. Neste contexto, Frantz (1985:56) acrescenta que a cooperativa também pode ser compreendida como a definição de uma “[...] estratégia de competição com o objetivo de maximizar os resultados da ação econômica individual de cada produtor [...]”. Esta pesquisa vislumbra as cooperativas de crédito na ótica do cooperativismo teórico e, partindo dos pressupostos e axiomas desenvolvidos, a análise da informação para tomada de decisões é condição central. Ferramentas e metodologias que visam à análise de informações gerenciais têm evoluído muito nas últimas décadas. A gestão de uma cooperativa de crédito é complexa, pois necessita manter o equilíbrio entre os anseios e necessidades dos cooperados e competir no mercado. As características de associação para os cooperados e de empresa para o mercado devem estar em certo equilíbrio. O número de cooperados e de cooperativas vêm aumentando paulatinamente. Segundo dados da OCB (2014), existem hoje no Brasil, 1.047 cooperativas de crédito singular e 4.529 pontos de atendimento. O SICOOB é o maior sistema de crédito

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

381

cooperativo do Brasil, congrega 529 cooperativas singulares e 1.949 pontos de atendimento cooperativo (Portal do Cooperativismo de Crédito, 2014). O ambiente dinâmico e competitivo do mercado financeiro brasileiro concomitante com mudanças na oferta de crédito nos últimos anos exige a adoção de uma postura profissional, o que conduz as cooperativas de crédito a adotarem o uso de novas tecnologias e conhecimentos gerenciais. Oliveira (2001) aponta a profissionalização de cooperados e de cooperativas como uma tendência relevante. O setor tem se desenvolvido de forma rápida, adotando uma estratégia de integração por meio de cooperativas centrais e necessitam, neste sentido, estar altamente em sintonia com o que há de mais eficiente em ferramentas de gestão. Analisar o crédito constitui certamente um dos pontos mais importantes em instituições financeiras. Chaia (2003) destaca a importância da definição do tipo de análise a ser feito e da abrangência da mesma e ainda alerta para o perigo de copiar e utilizar modelos de outras instituições, resultando assim em avaliações inadequadas. Um dos principais métodos de avaliação de crédito utilizado pelas instituições financeiras é o credit scoring. Chaia (2003, p. 23) define este modelo como o uso de ferramental estatístico na identificação dos fatores determinantes da probabilidade de o cliente tornar-se inadimplente, e aponta como principal vantagem o fato de que “[...] decisões sobre a concessão são tomadas com base em procedimentos impessoais e padronizados, gerando um maior grau de confiabilidade”. Com a concepção de cooperativa discutida anteriormente, torna-se altamente relevante o uso de tais metodologias objetivas na concessão de crédito, tal como o credit scoring. Evita-se que a decisão seja tomada somente pela avaliação em julgamentos subjetivos. Koh, Tan, e Goh (2006) condiciona o progresso do credit scoring ao aumento da competitividade, avanços na tecnologia computacional e no aumento exponencial de grandes bancos de dados. Mester (1997) indica que a exatidão do modelo, a atualização dos dados e a avaliação e readequação dos modelos são alguns fatores críticos do credit scoring. Falhas nesses fatores limitam o uso de tal modelo. Tendo em vista que a concessão de crédito é um dos processos centrais das cooperativas de crédito, a análise de tal processo caracteriza-se como ponto fundamental para proteger o patrimônio coletivo da cooperativa. A obtenção de ferramentas que classifiquem e ajudem a prever comportamentos de futuras concessões é fundamental para a gestão de crédito, com a vantagem de diminuir a subjetividade no processo, permitir a condução mais eficiente dos recursos e proporcionar maior celeridade nas propostas. Estudos de verificação de análise de crédito por meio de mineração de dados aumentam a precisão dos modelos e foram realizados por vários autores nos últimos anos (Abellán & Mantas, 2014; Akkoç, 2012; Bhattacharyya, Jha, Tharakunnel, & Christopher, 2011; Chang & Yeh, 2012; Chen & Huang, 2011; Crone & Finlay, 2012; Cubiles-De-La-Vega, Blanco-Oliver, Pino-Mejías, & Lara-Rubio, 2013; García, Marqués, & Sánchez, 2012; Han, Han, & Zhao, 2013; Koh et al., 2006; Kruppa,

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

www.jistem.fea.usp.br

382 Sousa, M. de M., Figueiredo, R. S.

Schwarz, Arminger, & Ziegler, 2013; Lai, Yu, Wang, & Zhou, 2006; Lemos, Steiner, & Nievola, 2005; Majeske & Lauer, 2013; Marqués, García, & Sánchez, 2012; Nie, Rowe, Zhang, Tian, & Shi, 2011; Oreski & Oreski, 2014; Saberi et al., 2013; Wang, Ma, Huang, & Xu, 2012; Xiong, Wang, Mayers, & Monga, 2013; Yap, Ong, & Husain, 2011; Zhong, Miao, Shen, & Feng, 2014; Zhou, Jiang, Shi, & Tian, 2011; Zhu, Li, Wu, Wang, & Liang, 2013). Apesar do crescente interesse, a aplicação dessas ferramentas em cooperativas ainda é pouco realizado. Khatchatourian e Treter (2010) aplicaram lógica Fuzzy na análise do desempenho financeiro em cooperativas de produção do Rio Grande do Sul. Zhu, Li, Wu, Wang, e Liang (2013) utilizaram Support vector machine - máquina de vetores de suporte – na análise de crédito em cooperativa de crédito de Barbados. Há atualmente diversas técnicas de Mineração de Dados disponíveis. Assim, pretendeu-se examinar qual metodologia de mineração oferece melhores resultados na análise de crédito para Cooperativas de Crédito. Neste sentido, indaga-se, se um modelo de Mineração de Dados pode ter bom desempenho na classificação e previsão na gestão de crédito em cooperativas de crédito. 2. REFERENCIAL TEÓRICO A terminologia “Descoberta de Conhecimento em Base de Dados” (Knowledge Discorery in Databases – KDD) foi utilizada pela primeira vez em 1989 para destacar que o conhecimento é o produto final do processo de descoberta em base de dados (Fayyad, Piatetsky-Shapiro, & Smyth, 1996). Os termos KDD e Mineração de Dados foram entendidos por muitos pesquisadores como sinônimos até 1995 (Lemos et al., 2005). Fayyad et al. (1996) conceitua e distingue KDD e Mineração de Dados da seguinte forma: o primeiro referese ao processo geral de descobrir conhecimento útil dos dados e o segundo à aplicação específica de algoritmos para a extração de padrões e modelos dos dados. No conceito destes autores, Mineração de Dados seria, então, um passo no processo de KDD, consistindo de empregar análise de dados e algoritmos na produção de um conjunto particular de padrões e modelos. Fayyad et al. (1996) denomina padrões como componentes dos modelos. Neste estudo foi utilizado o conceito de modelo definido por Pidd (1998, p. 23): “Modelo é uma representação externa e explícita de parte da realidade vista pela pessoa que deseja usar aquele modelo para entender, mudar, gerenciar e controlar parte daquela realidade”. Goldschmidt e Passos (2003, p. 6) dividem as atividades de KDD em três grupos: (i) desenvolvimento tecnológico – este grupo compreende “[...] as iniciativas de concepção, aprimoramento e desenvolvimento de algoritmos, ferramentas e tecnologias de apoio [...]” no processo de KDD; (ii) execução de KDD – este grupo inclui atividades relacionadas à utilização dos algoritmos, ferramentas e tecnologias desenvolvidas na procura de conhecimento; (iii) aplicação de resultados – com os modelos desenvolvidos na execução de KDD, “[...] as atividades se voltam à aplicação dos resultados no contexto em que foi realizado o processo de KDD”.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

383

É comum a comparação de técnicas e modelos híbridos ou compostos. Dentre os modelos de análise de crédito e risco destacam-se o uso de: (a) regressão logística (Akkoç, 2012; Bhattacharyya et al., 2011; Cubiles-De-La-Vega et al., 2013; Han et al., 2013; Ju & Sohn, 2014; Koh et al., 2006; Kruppa et al., 2013; Nie et al., 2011; Wang et al., 2012; Yap et al., 2011); (b) árvores de decisão (Abellán & Mantas, 2014; Bhattacharyya et al., 2011; Chen & Huang, 2011; Crone & Finlay, 2012; Koh et al., 2006; Kruppa et al., 2013; Lemos et al., 2005; Nie et al., 2011; Wang et al., 2012; Yap et al., 2011); (c) redes neurais (Akkoç, 2012; Chen & Huang, 2011; Koh et al., 2006; Lai et al., 2006; Nie et al., 2011; Oreski & Oreski, 2014; Saberi et al., 2013; Wang et al., 2012); (d) Support vector machine - máquina de vetores de suporte (Bhattacharyya et al., 2011; Nie et al., 2011; Xiong et al., 2013; Zhong et al., 2014; Zhu et al., 2013); (e) métodos ensemble (Abellán & Mantas, 2014; García et al., 2012; Marqués et al., 2012; Nie et al., 2011; Wang et al., 2012). O contexto real de aplicação dos estudos foram coletados em organizações do Canadá (Xiong et al., 2013), Alemanha (Han et al., 2013; Koh et al., 2006), Croácia (Oreski & Oreski, 2014), Peru (Cubiles-De-La-Vega et al., 2013), China (Nie et al., 2011), Turquia (Akkoç, 2012) e Barbados (Zhu et al., 2013). Análise de crédito por meio de mineração de dados ainda é escassa no Brasil, Lemos et al. (2005) verificaram a aplicação da análise de crédito bancário com a mesma metodologia e utilizaram como lócus uma agência do Banco do Brasil. As árvores de decisão constituem um dos principais e mais populares métodos de Mineração de Dados (Wang et al., 2012). Este método, conforme Lemos et al. (2005, p. 229) é o único a exibir resultados em forma hierárquica, “[...] o atributo mais importante é apresentado na árvore como o primeiro nó, e os atributos menos relevantes são mostradas nos nós subsequentes”. Assim, árvore de decisão é uma estrutura usada para dividir grande quantidade de dados em sucessivos conjuntos menores pela aplicação de uma sequência de regras de decisão (Berry & Linoff, 2004). A construção de árvores de decisão é especialmente atrativa no ambiente de KDD. As causas para tal propensão, abordadas por Gehrke (2003), são: resultado intuitivo e de fácil entendimento; árvores de decisão são não-paramétricas, aplicáveis, portanto, a tratamentos exploratórios; construção relativamente rápida comparada a outros métodos; a acurácia da árvore de decisão pode ser comparada com outros modelos. É comum a transformação de uma árvore de decisão em regras de decisão. Árvore de Decisão pode ser compreendida como: [...] um grafo em que cada nó não folha representa um predicado (condição) envolvendo um atributo e um conjunto de valores. Os nós da folha correspondem à atribuição de um valor ou conjunto de valores a um atributo do problema (Goldschmidt & Passos, 2005, p. 57). Neste sentido, os caminhos da árvore correspondem a regras do tipo “SE <condições> ENTÃO <conclusão>”. Há muitos algoritmos desenvolvidos baseados na

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

www.jistem.fea.usp.br

384 Sousa, M. de M., Figueiredo, R. S.

indução de árvores de decisão, dentre os quais se destacam o C4.5, o CART (Classification and Regression Trees1), QUEST (Quick, unbiased, efficient statistical tree2) e CHAID (chi-square automatic interaction detector3). Rede Neural Artificial – RNA - é um modelo matemático baseado na estrutura cerebral, ordenado em camadas e ligações. As RNAs têm origem em 1943, entretanto, é na década de 1980 que é despertado maior interesse pelo método, tendo como principal fator de desenvolvimento o avanço da tecnologia da informação (Braga, Carvalho, & Ludermir, 2000). Na perspectiva de Goldschmidt e Passos (2005, p. 175), RNAs podem ser compreendidas como “[...] modelos matemáticos inspirados nos princípios de funcionamento dos neurônios biológicos e na estrutura do cérebro”. Tais modelos, conforme os mesmos autores permitem simular capacidades humanas de aprender, generalizar, associar e abstrair. Braga et al. (2000, p. 1) conceituam RNAs como “sistemas paralelos distribuídos compostos por unidades de processamento simples (nodos) que calculam determinadas funções matemáticas (normalmente não-lineares), [...] dispostas em uma ou mais camadas e interligadas por um grande número de conexões[...]” A estrutura de uma RNA é, portanto, composta de camadas de neurônios e conexões, que são ponderadas por pesos. Conforme a Figura 01, os neurônios são representados pelos nodos e os pesos são representados pelas setas.

Figura 01 – Arquitetura de uma RNA. Fonte: Goldschmidt & Passos (2005). 1 2 3

Árvores de classificação e regressão. Árvore estatística eficiente, rápida, sem viés. Detector interativo automático qui-quadrado.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

385

Há tipicamente três partes no processamento de RNA: a camada de entrada ou input layer, por onde são recebidos os dados; a camada interna, comumente chamada de “camada escondida” ou hidden layer, responsável pelo processamento dos dados, esta parte pode conter mais de uma camada e a camada de saída ou output layer, representando o resultado (Larose, 2005). O primeiro passo da aplicação de um RNA é a etapa de aprendizagem da rede, onde há o ajuste dos parâmetros. Este aprendizado pode ser classificado em duas categorias: supervisionado e não-supervisionado, o primeiro ocorre quando é fornecido variáveis de saída, o segundo não necessita da variável alvo. Braga et al. (2000, p. 227) indicam como pontos positivos que, suscitam interesse pelo método, a habilidade de aprendizado e posterior generalização, com a possibilidade de mapear funções multivariadas, a auto-organização, o processo de séries temporais, a possibilidade do uso de grande número de variáveis de entrada, a possibilidade do uso de amostragens e por ser caracterizada como um modelo nãoparamétrico, portanto, e ainda ressaltam que, “[...] não há grande necessidade de se entender o processo propriamente dito”. Entretanto, este último aspecto é também considerado pelos mesmos autores como a principal crítica, ou seja, a inabilidade do modelo em esclarecer de que maneira os resultados são gerados. Devido a esta especificidade, as RNAs são também denominadas de “caixas pretas”. 3.

METODOLOGIA

Para a realização da pesquisa optou-se pelo estudo de caso que, segundo Yin (2010), é oportuno para estudar acontecimentos contemporâneos em um contexto da vida real, quando o controle se torna mais difícil para o pesquisador. O estudo de caso é caracterizado como do tipo único, contemplando uma unidade de análise, envolvendo uma Cooperativa de Crédito do sistema SICOOB, estruturado de forma a contemplar a investigação da Cooperativa sob o enfoque do cooperativismo teórico. A base de dados da cooperativa foi utilizada para avaliar o desempenho do sistema de análise de crédito. Essa base de dados corresponde aos dados históricos das análises de pessoas físicas de 2003 a 2007. Devido à mudança no sistema de informações, não é possível coleta dos dados anteriores a este período. Os dados referentes à análise de crédito são altamente confidenciais e estratégicos devido ao sigilo bancário e também ao risco dos concorrentes adquirirem tais dados, torna-se, portanto, muito difícil de serem adquiridos por terceiros. A opção pela cooperativa foi, então, por sua disponibilidade em fornecer os dados. Atualmente, a cooperativa utiliza como ferramenta para realizar sua análise de crédito, um aplicativo do sistema SICOOB chamado SisBr. É esse aplicativo que contém as informações que a gerência e diretoria se valem para fundamentar suas decisões de conceder ou não crédito. A pesquisa utilizou os passos sugeridos por Fayyad, Piatetsky-Shapiro, e Smyth (1996) no processo de Descoberta de Conhecimento: seleção dos dados; pré-

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

www.jistem.fea.usp.br

386 Sousa, M. de M., Figueiredo, R. S.

processamento e limpeza dos dados; transformação dos dados; Mineração de Dados; interpretação e avaliação dos resultados. Para a consecução dos objetivos, este estudo se apoiou nas atividades de execução de KDD, conforme discutido por Goldschmidt e Passos (2005). Dentre as técnicas de Mineração de Dados disponíveis, foram utilizadas Redes Neurais e Árvores de Decisão, ambas encontradas extensivamente nos estudos empíricos. A coleta e seleção dos dados correspondem ao processo de captar, organizar e selecionar os dados disponíveis para a etapa da modelagem e Mineração de Dados, portanto, requer exame acurado. Dasu e Johnson (2003) apontam os seguintes elementos auxiliadores na análise de variáveis: a experiência anterior, o conhecimento, a quantidade e a qualidade dos dados. 3. RESULTADOS Nesta parte do trabalho encontram-se os resultados das simulações das técnicas investigadas neste artigo, Árvore de Decisão e Redes Neurais Artificiais, bem como o teste estatístico comparativo entre ambas. Não foram encontrados valores ausentes na base de dados estudada. O Quadro 01 apresenta a estrutura da base de dados construída, com as variáveis e valores possíveis. Quadro 01 - Estrutura da base de dados. CÓDIGO 01

VARIÁVEIS Código do cooperado

Gênero

Idade

Nível educacional

05 06

Cidade Naturalidade

Local de domicílio

Principal atividade

Estado civil

Capital

Relacionamento

VALOR Chave numérica, única para cada cooperado. 1= masculino 2= feminino Valor numérico 1= pós-graduação 2= superior completo 3= superior incompleto 4= segundo grau completo 5= segundo grau incompleto 6= primeiro grau completo 7= primeiro grau incompleto Nome da cidade onde mora Nome da cidade onde nasceu. 1= zona urbana 2= zona rural Nome da atividade 1= casado 2= solteiro 3= viúvo 4= separado judicialmente 5= outros Valor numérico 1= opera com a cooperativa há mais de 3 anos 2= opera com a cooperativa de 1 a 3 anos 3= opera com a cooperativa até 1 ano

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

387

Quadro 01 - Estrutura da base de dados. (cont.) CÓDIGO

VARIÁVEIS

Comportamento das operações

Tempo de experiência na atividade/emprego

Consultas cadastrais

Informações cadastrais na cooperativa

Finalidade da operação

Garantia das operações

Liquidez das garantias

Que frequência o cooperado opera (operações ativas)

Valor da operação

Nível de comprometimento – prestações com renda Líquida do cooperado Patrimônio líquido pessoal livre de relacionamento com endividamento total

Endividamento total em relação à renda líquida ano

Total endividamento em relação ao capital integralizado

Perfil da atividade econômica do associado

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

VALOR 1= normal 2= atrasos esporádicos 3= atrasos/renegociações constantes 1= mais de 5 anos 2= de 2 a 5 anos 3= até 3 anos 1= ausência de restrições 2= com restrições irrelevantes justificadas 3= com restrições relevantes ou irrelevantes sem justificativas 1= cadastro atualizado e confiável 2= cadastro atualizado e não confiável 3= informações desatualizadas ou ausência de informações 1= custeio e investimento 2= financiamento de bens 3= crédito pessoal/cheque especial 4= renovações/composições de dívidas 1= hipoteca – capital social 2= alienação fiduciária/warrants 3= penhor cedular/caução de títulos 4= pessoal 1= garantia de fácil liquidez (venda até 6 meses) 2= garantia de média liquidez (venda de 6 a 12 meses) 3= garantia pessoal ou de difícil liquidez (venda com prazo maior de 12 meses) 1= nunca 2= frequentemente 3= permanentemente 1= até 1% do Patrimônio Líquido Ajustado (PLA) 2= de 1,01% a 2% do PLA 3= de 2,01% a 3% do PLA 4= mais de 3% do PLA 1= até 20% da renda média líquida 2= de 20% a 30% da renda média líquida 3= mais de 30% da renda líquida 1= mais de 4 vezes 2= de 2 a 4 vezes 3= sem patrimônio pessoal ou até 2 vezes 1= até 2 vezes 2= de 2 a 4 vezes 3= mais de 4 vezes 1= até 4 vezes 2= de 4 a 8 vezes 3= de 8 a 12 vezes 4= acima de 12 vezes 1= ótima 2= boa 3= regular 4= ruim

www.jistem.fea.usp.br

388 Sousa, M. de M., Figueiredo, R. S.

Quadro 01 - Estrutura da base de dados (Cont.) CÓDIGO

VARIÁVEIS

Risco atribuído pela cooperativa

Resultado-Julho 2007

Resultado-Agosto 2007

Resultado-Setembro 2007

Resultado-Outubro 2007

Resultado-Novembro 2007

Resultado-Dezembro 2007

Resultado-Janeiro 2008

Resultado-Fevereiro 2008

Resultado-Março 2008

Resultado-Abril 2008

Resultado-Maio 2008

Resultado-Junho 2008

Resultado agregado

VALOR 1= AA 2= A 3= B 4= C 5= D 6= E 7= F 8= G 9= H 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente 1= adimplente 2= inadimplente

Fonte: Dados da pesquisa, 2010.

A variável de saída é representada pelas variáveis 27 a 39, correspondentes ao período de julho de 2007 a junho de 2008. As variáveis 02 a 10 não constam na análise de crédito realizada pela cooperativa, assim, foram agregadas com a finalidade de ampliar a análise. A coleta destas variáveis foi realizada nos cadastros dos cooperados e representam os dados disponíveis pela cooperativa. As variáveis 11 a 26 são utilizadas atualmente na análise de crédito e representam o comportamento histórico do cooperado na tomada de crédito. Os códigos 27 a 39 representam a variável de saída adotada pela pesquisa e retratam o período de julho de 2007 a junho de 2008. Estes são os dados disponibilizados pela cooperativa. Dados anteriores a estes não estão disponíveis.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

389

A quantidade de variáveis utilizadas na análise enquadra-se dentro do utilizado em outras pesquisas. Por exemplo, Koh et al. (2006) utilizaram 20 variáveis e Lemos et al. (2005) utilizaram 24. A variável, código do cooperado foi descartada, pois, foi útil somente para identificar o cooperado na coleta dos dados. A variável “risco atribuído” foi utilizada somente na etapa de pré-processamento e limpeza dos dados. Não foi utilizada nas etapas de transformação e modelagem porque esta variável se refere à saída do modelo utilizado pela cooperativa, portanto, representa o resultado do modelo atualmente empregado. A variável, resultado agregado, representa o resultado do período analisado (julho de 2007 a junho de 2008) e, segundo as regras do negócio da cooperativa e dos objetivos de estudo, constitui a variável-alvo de saída do modelo. A transformação dos dados visa auxiliar a execução das técnicas de mineração de dados. Os dados foram agrupados segundo indicação de Goldschmidt & Passos (2005) em uma única tabela bidimensional. Os dados foram coletados de duas fontes na cooperativa: da avaliação de crédito e do cadastro, de forma manual, registro a registro. Para a pesquisa foram utilizados dados históricos de 211 cooperados, pessoas físicas, sendo 22 inadimplentes e 189 adimplentes. Os dados representam o universo dos cooperados tomadores de crédito. Dado o desbalanceamento entre inadimplentes e adimplentes, é possível incorrer em viés e problema de superajustamento (Chawla, 2005; Horta, Borges, Carvalho, & Alves, 2011). Para solucionar tal problema foi utilizada a técnica denominada SMOTE (Synthetic Minority Oversampling Technique) (Chawla, 2005) para inclusão de observações de cooperados inadimplentes. Esse algoritmo é considerado um dos mais utilizados pela literatura (Horta et al., 2011). Dessa forma foram criadas 110 observações da classe minoritária, ou seja, inadimplentes. A base ficou com 132 cooperados inadimplentes e 189 adimplentes, constituindo uma amostra de 321 observações. Posteriormente, a base de dados foi randomizada para evitar a concentração de mesmos valores em determinado conjunto de dados na validação cruzada e incorrer em superajustamento. Para a implementação computacional das técnicas Árvore de Decisão e Redes Neurais, foi utilizada a base de dados descritos neste estudo, considerando-se para cada cooperado as variáveis já descritas anteriormente. Para exemplificar, foi escolhido um modelo gerado pela técnica Árvore de Decisão, para transformar em regras de decisão na fase pós-processamento. Optou-se por utilizar a ferramenta computacional de domínio público WEKA (Waikato Environment for Knowledge Analysis). Goldschmidt e Passos (2005, p. 50) argumentam que, para melhor fidedignidade da avaliação do modelo de conhecimento, “[...] os dados utilizados na construção do modelo não devem ser os mesmos utilizados na avaliação desse modelo”. Os mesmos autores ainda afirmam que deve haver no mínimo duas partições: a partição de treinamento e a partição de teste. A primeira inclui os dados para a construção do modelo e a segunda, os dados para avaliação.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

www.jistem.fea.usp.br

390 Sousa, M. de M., Figueiredo, R. S.

Dividir o conjunto de dados tem o propósito de simplificar, sumarizar e reduzir a variabilidade e tamanho da base de dados, resultando na seleção de modelos mais sofisticados e acurados (Dasu & Johnson, 2003). Neste estudo, para melhor isenção da avaliação, tanto para a árvore de decisão quanto para a RNA, foi utilizada a validação cruzada com K conjuntos (K-Fold CrossValidation). Segundo Goldschmidt e Passos (2005, p. 51), neste método, a base de dados é dividida aleatoriamente com N elementos em K subconjuntos separados, “cada um dos K subconjuntos é utilizado como conjunto-teste e os (K-1) demais subconjuntos são reunidos em um conjunto de treinamento. O processo é repetido K vezes, sendo gerados e avaliados K modelos [...]”. Os dados foram divididos em dez conjuntos e repetidos em dez simulações conforme proposto por Witten e Frank (2005). A validação cruzada é encontrada em vários estudos de análise de crédito (Akkoç, 2012; Chang & Yeh, 2012; Han et al., 2013) Para este estudo, decidiu-se por utilizar RNA de múltiplas camadas, Multilayer Perceptron (MLP), com o algoritmo de aprendizagem backpropagation. A quantidade de neurônios da camada de entrada foi de 66, da camada intermediária 2 e a quantidade de neurônios na camada de saída foi igual a 2. Em todos os testes foi utilizada taxa de aprendizagem igual a 0.01, dado que segundo as simulações foi a melhor taxa que melhorou a classificação e também foi utilizada por Lemos et al., (2005). Optou-se por não utilizar a taxa momentum conforme Lemos et al. (2005). Além disso, o acréscimo dessa taxa não melhorou o desempenho da classificação. O aprendizado da RNA é do tipo supervisionado. Ferreira (2005, p. 37) descreve este tipo da seguinte forma: “[...] a rede é treinada através do fornecimento dos valores de entrada e dos respectivos valores de saída [...]”. Para a avaliação comparativa dos modelos foi utilizado o parâmetro percentual total de valores preditos corretamente com o teste estatístico t modificado (corrected resample t-test) com nível de significância de 0.05 (ou 5%) em duas caudas e nove graus de liberdade, conforme proposto por Witten e Frank (2005) de acordo com a fórmula 1, apresentada abaixo:

(1) Em que:

diferença das médias de acerto de valores preditos entre os modelos desenvolvidos. número de conjuntos vezes o número de repetições. número de amostras para teste. número de amostras para treinamento. variância da diferença das médias.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

391

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

3.1 Árvore de Decisão Optou-se, nesta pesquisa, por utilizar a ferramenta J4.8, que é a implementação do software WEKA do algoritmo da árvore de decisão C4.5. Segundo Goldschmidt e Passos (2005), esta árvore é amplamente utilizada e aceita. Tomou-se um modelo gerado pela técnica Árvore de Decisão para exemplificar as regras e matriz de confusão. O modelo abaixo gerou 41 folhas, ou seja, conjuntos de regras de decisão do tipo se-então. Algumas regras do primeiro conjunto estão apresentadas abaixo.  Se Liquidez das garantias = garantia de fácil liquidez (venda até 6 meses) então inadimplente.  Se Liquidez das garantias = garantia de média liquidez (venda de 6 a 12 meses) e Nível de comprometimento = até 20% da renda média líquida então adimplente.  Se Liquidez das garantias = garantia de média liquidez (venda de 6 a 12 meses) e Nível de comprometimento = de 20% a 30% da renda média líquida então adimplente. O Quadro 02 mostra a matriz de confusão gerada pelo conjunto de teste da árvore avaliada. Esta matriz apresenta as instâncias classificadas em previstos e reais para avaliar o tipo de acerto e erro dos modelos. A diagonal principal indica os valores corretamente classificados. Os valores estão expressos em termos absolutos. Quadro 02: Matriz de confusão de um modelo desenvolvido pelo método Árvore de Decisão. Previstos Inadimplente Reais

Inadimplente Adimplente

Adimplente 121

181

Fonte: Dados da pesquisa, 2010.

Neste conjunto, o modelo baseado no algoritmo C4.5 de árvore de decisão classificou 302 registros corretamente, representando assim uma taxa de acerto de 94,08% e 19 observações incorretas, representando 5,92%. 3. 2. Rede Neural A RNA deste estudo, conforme já discutido no referencial teórico, foi constituída por três camadas: entrada, intermediária e saída. O aprendizado da rede foi do tipo supervisionado, pois foram indicados os valores de saída do modelo. Da mesma forma que a Árvore de Decisão, foi tomado um modelo para exemplificar os resultados gerados pelo software WEKA. O Quadro 03 apresenta a matriz de confusão para um modelo desenvolvido.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

www.jistem.fea.usp.br

392 Sousa, M. de M., Figueiredo, R. S.

Quadro 03 – Matriz de confusão de um modelo desenvolvido pelo método RNA. Previstos Inadimplente Reais

Adimplente

Inadimplente

118

Adimplente

176

Fonte: Dados da pesquisa, 2010.

Este modelo baseado em RNA, com o uso do algoritmo Multilayer Perceptrom, classificou 294 registros corretamente, representando uma taxa de acerto de 91,59% e 27 observações incorretas, representando 8,41%. 3. 3. Avaliação dos Modelos Nesta seção são avaliados comparativamente os modelos desenvolvidos neste estudo: RNA (algoritmo MultilayerPerceptrom) e Árvore de Decisão (algoritmo C4.5). Para a realização desta avaliação, foi utilizado o percentual total de valores preditos corretamente. A Tabela 01 apresenta o resultado do percentual dos valores preditos corretamente e o respectivo desvio padrão das simulações feitas com os modelos em estudo. Tabela 01 – Avaliação comparativa. Algoritmo

C4.5

MultilayerPerceptrom

Percentual correto

97,07%

95,58%

Desvio padrão

2,76

3,47

Fonte: Dados da pesquisa, 2010.

As simulações feitas na ferramenta Experimenter do WEKA indicam que a técnica Árvore de Decisão, com a implementação do algoritmo C4.5, é estatisticamente semelhante ao nível de significância de 0.05 em duas caudas que a RNA com a implementação do algoritmo MultilayerPerceptrom. A Figura 02 apresenta a saída do software WEKA das simulações feitas do problema previamente descrito.

Figura 02: Caixa de saída das simulações da ferramenta experimenter do WEKA. Fonte: Dados da pesquisa, 2010.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

393

O desempenho da árvore de decisão para o presente problema foi superior ao encontrado por Yap et al. (2011) que obteve uma taxa de erro de 28,1%. O estudo de Lemos et al. (2005), apesar de não realizar teste estatístico, encontrou taxa de acerto maior da rede neural em comparação com a árvore de decisão. Apesar da semelhança estatística, as árvores de decisão são consideradas de fácil uso (Lemos et al., 2005). Os resultados indicam que os modelos de classificação por meio de mineração de dados desenvolvidos podem ser úteis na avaliação pela cooperativa e, dessa forma, melhorar o desempenho, conforme já encontrado na análise em organização de microcrédito (Cubiles-De-La-Vega et al., 2013) e cooperativa de crédito (Zhu et al., 2013). 4. CONCLUSÕES O objetivo deste estudo foi desenvolver e avaliar modelos de Mineração de Dados para classificar e prever o comportamento dos cooperados em saldar os compromissos contraídos. Para o desenvolvimento do modelo foram utilizadas Árvore de Decisão e RNA, ambas, técnicas de Mineração de Dados. O processo de preparação e modelagem dos dados seguiu os passos sugeridos pela literatura: seleção dos dados; pré-processamento e limpeza dos dados; transformação dos dados; Mineração de Dados; interpretação e avaliação dos resultados. Os dados foram particionados em conjuntos de treinamento e teste. Embora a Árvore de Decisão tenha obtido 97,07% de acerto nas simulações feitas e a RNA 95,58%, o algoritmo baseado em Árvore de Decisão C4.5 obteve resultado estatisticamente semelhante ao modelo baseado em Redes Neurais Artificiais MultilayerPerceptrom. O processo de descoberta de conhecimento e o uso dos modelos baseados em Mineração de Dados desenvolvidos podem trazer vantagens práticas para a cooperativa. A compreensão das variáveis e seus relacionamentos podem ajudar a melhor classificar e prever o comportamento dos cooperados. A avaliação mais profunda das variáveis pode ainda ajudar a incluir variáveis que sejam importantes e excluir variáveis que não se mostrem relevantes, com vantagens de proporcionar modelos de gestão de crédito mais sucintos e precisos, com economia de tempo na execução e melhor acurácia nas decisões. A análise de casos discrepantes, ou que estão fora do padrão, pode ser relevante para a formação de uma nova classificação ou, inversamente, compor padrões indesejados. Pode-se enumerar como limitação deste trabalho: a ausência de bases de dados de outras cooperativas, para comparar, avaliar e validar o modelo; limitações do sistema de informações utilizado pela cooperativa que impossibilita a coleta de dados das variáveis de entrada anteriores a 2003 e limita a fornecer os dados referentes à variável de saída há apenas seis meses; a falta de integração de alguns módulos da base de dados com planilhas eletrônicas.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

www.jistem.fea.usp.br

394 Sousa, M. de M., Figueiredo, R. S.

Sugerem-se as seguintes propostas para estudos posteriores: a utilização de diferentes bases de dados para validação do modelo de análise de crédito; o uso de outras técnicas de Mineração de Dados; o uso de modelos híbridos com a combinação de diferentes técnicas para aperfeiçoar e melhorar o desempenho na classificação e previsão; análises de investimentos com a avaliação do tipo de erro e qual o impacto financeiro que o modelo apresenta para a lucratividade e rentabilidade da cooperativa; avaliação dos casos discrepantes, principalmente para a variável capital, com o objetivo de verificar a existência de novos padrões e classificações. REFERÊNCIAS Abellán, J., & Mantas, C. J. (2014). Improving experimental studies about ensembles of classifiers for bankruptcy prediction and credit scoring. Expert Systems with Applications, 41(8), 3825–3830. Akkoç, S. (2012). An empirical comparison of conventional techniques, neural networks and the three stage hybrid Adaptive Neuro Fuzzy Inference System (ANFIS) model for credit scoring analysis: The case of Turkish credit card data. European Journal of Operational Research, 222(1), 168–178. Berry, M. J. A., & Linoff, G. (2004). Data mining techniques: For marketing, sales and customer relationship management (2nd ed.). Indianapolis: wiley Publishing. Bhattacharyya, S., Jha, S., Tharakunnel, K., & Christopher, J. (2011). Data mining for credit card fraud: A comparative study. Decision Support Systems, 50(3), 602–613. Braga, A. de P., Carvalho, A. P. de L. F., & Ludermir, T. B. (2000). Redes neurais artificiais: Teoria e aplicações. Rio de Janeiro: LTC. Chaia, A. J. (2003). Modelos de gestão de risco de crédito e sua aplicabilidade ao mercado brasileiro. Dissertação de Mestrado. FEA/USP. Chang, S.-Y., & Yeh, T.-Y. (2012). An artificial immune classifier for credit scoring analysis. Applied Soft Computing, 12(2), 611–618. Chawla, N. V. (2005). Data mining for imbalanced datasets: An overview. In Data mining and knowledge discovery handbook (pp. 853–867). New Jersey: Springer. Chen, S. C., & Huang, M. Y. (2011). Constructing credit auditing and control & management model with data mining technique. Expert Systems with Applications, 38(5359-5365). Crone, S. F., & Finlay, S. (2012). Instance sampling in credit scoring: An empirical study of sample size and balancing. International Journal of Forecasting, 28(1), 224– 238. Cubiles-De-La-Vega, M.-D., Blanco-Oliver, A., Pino-Mejías, R., & Lara-Rubio, J. (2013). Improving the management of microfinance institutions by using credit scoring models based on Statistical Learning techniques. Expert Systems with Applications, 40(17), 6910–6917. Dasu, T., & Johnson, T. (2003). Exploratory data mining and data cleaning. New Jersey: John Wiley & Sons. Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). From data mining to knowledge discovery in databases. A I Magazine, 17(3), 37–54.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

Análise de Crédito por Meio de Mineração de Dados: Aplicação em Cooperativa de Crédito

395

Ferreira, J. B. (2005). Mineração de dados na retenção de clientes em telefonia celular. Dissertação de Mestrado. PUC-RIO. García, V., Marqués, A. I., & Sánchez, J. S. (2012). On the use of data filtering techniques for credit risk prediction with instance-based models. Expert Systems with Applications, 39(18), 13267–13276. Gehrke, J. (2003). Decision tree. In The handbook of data mining (pp. 3–23). New Jersey: Lawrence Erlbaum Associates. Goldschmidt, R., & Passos, E. (2005). Data mining: Um guia prático. Rio de Janeiro: Elsevier. Han, L., Han, L., & Zhao, H. (2013). Orthogonal support vector machine for credit scoring. Engineering Applications of Artificial Intelligence, 26(2), 848–862. Horta, R. A. M., Borges, C. C. H., Carvalho, F. A. A., & Alves, F. J. S. (2011). Previsão de insolvência: Uma estratégia para balanceamento da base de dados utilizando variáveis contábeis de empresas brasileiras. Sociedade, Contabilidade E Gestão, 6(2), 21–36. Ju, Y. H., & Sohn, S. Y. (2014). Updating a credit-scoring model based on new attributes without realization of actual data. European Journal of Operational Research, 234(1), 119–126. Khatchatourian, O., & Treter, J. (2010). APLICAÇÃO DA LÓGICA FUZZY PARA AVALIAÇÃO ECONÔMICO-FINANCEIRA DE COOPERATIVAS DE PRODUÇÃO. Revista de Gestão Da Tecnologia E Sistemas de Informação, 7(1), 141– 162. Koh, H. C., Tan, W. C., & Goh, C. P. (2006). A two-step method to construct credit scoring models with data mining techniques. International Journal of Business and Information, 1(1), 96–118. Kruppa, J., Schwarz, A., Arminger, G., & Ziegler, A. (2013). Consumer credit risk: Individual probability estimates using machine learning. Expert Systems with Applications, 40(13), 5125–5131. Lai, K. K., Yu, L., Wang, S., & Zhou, L. (2006). Credit risk analysis using a reliabilitybased neural network ensemble model. In Artificial Neural Networks-ICANN 2006 (pp. 682–690). Springer Berlin Heidelberg. Larose, T. D. (2005). Discovering knowledge in data: An introduction to data mining. New Jersey: John Wiley & Sons. Lemos, E. P., Steiner, M. T. A., & Nievola, J. C. (2005). Análise de crédito bancário por meio de redes neurais e árvore de decisao: Uma aplicação simples de data mining. Revista de Administração Da Universidade de São Paulo, 40(3), 225–234. Majeske, K. D., & Lauer, T. W. (2013). The bank loan approval decision from multiple perspectives. Expert Systems with Applications, 40(5), 1591–1598. Marqués, A. I., García, V., & Sánchez, J. S. (2012). Two-level classifier ensembles for credit risk assessment. Expert Systems with Applications, 39(12), 10916–10922. Mester, L. J. (1997). What’s the point of credit scoring? Business Review, 3, 3–16. Nie, G., Rowe, W., Zhang, L., Tian, Y., & Shi, Y. (2011). Credit card churn forecasting by logistic regression and decision tree. Expert Systems with Applications, 38(12), 15273–15285.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.379-396

www.jistem.fea.usp.br

396 Sousa, M. de M., Figueiredo, R. S.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 379-396

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 397-414 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200010

PERFORMANCE MEASUREMENT OF INFORMATION TECHNOLOGY GOVERNANCE IN BRAZILIAN FINANCIAL INSTITUTIONS Sara C. Boni Barbosa Ildeberto Aparecido Rodello Silvia Inês Dallavalle de Pádua University of Sao Paulo, USP/FEARP, Ribeirão Preto, SP, Brazil

_____________________________________________________________________ ABSTRACT Information technology governance is a process by which organizations align their information technology operations and services with their performance goals and strategic objectives and assess the results. The financial sector has benefited greatly from the development of information technology, primarily in bank automation. There are few research works aimed at describing information technology governance, and there are even fewer in the financial sector. This paper is aimed at measuring information technology governance in financial institutions using the methodology proposed by Weill and Ross (2004). The sample consisted of sixty professionals from financial institutions operating in Brazil. The average performance score was approximately 79.6, showing that the organizations studied have not achieved maximum performance (100), although many are very close to it. This paper describes an original empirical study of Brazilian financial institutions concerning information technology management performance. The survey investigates not only the context of performance, but also the importance that the organization places on each item related to information technology governance aligned with business needs and goals. Keywords: Information technology governance; Information technology; Corporate governance; Financial institutions. _____________________________________________________________________________________________ Manuscript first received/Recebido em: 15/08/2012 Manuscript accepted/Aprovado em: 30/04/2014 Address for correspondence / Endereço para correspondência Sara C. Boni Barbosa is a project management specialist at CIELO S.A. She has a bachelor’s degree in Business Administration from the University of Sao Paulo College of Economics, Business and Accountancy at Ribeirão Preto (FEARP) and is a post-graduate student of project management at Berkeley University. E-mail: sara_cbb@hotmail.com Ildeberto Aparecido Rodello is a professor of Information Systems at the University of São Paulo (USP) College of Economics, Business and Accountancy at Ribeirão Preto (FEARP). He has a Ph.D. in Applied Sciences (Virtual Reality) and develops research on topics such as Information Systems, Open Source ERP Systems and Distance Learning. He lectures on information systems development, e-commerce, distance learning and organizational change through information technology. E-mail: rodello@fearp.usp.br Silvia Inês Dallavalle de Pádua is a professor of Information Systems at the University of Sao Paulo College of Economics, Business and Accountancy at Ribeirão Preto (FEARP). She has a Ph.D. in Engineering and develops research on topics such as Business Process Management and Information Systems. She lectures on information system development and business process management. E-mail: dallavalle@fearp.usp.br University of São Paulo – Business Administration Department – College of Economics, Business and Accountancy at Ribeirão Preto (FEARP/USP). Address: Avenida dos Bandeirantes, 3900, Monte Alegre, Ribeirão Preto – SP – Brazil – 14040-900 Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

398 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

INTRODUCTION

Obtaining accurate information as soon as possible is recognized by organizations as an important tool for competitive survival and is considered one of the most important strategic resources (Weill & Ross, 2004). In the global context of rapid changes and fast communication, information has become a strategic asset, and information technology (IT) is an important contributor to the success of the economy (Afzali et al., 2010; Ayat, Masrom, & Sahibuddin, 2011; Krey et al., 2011). According to Khadra et al. (2009), enterprises understand the growing importance of IT and consider it a treasure in enhancing their competitive position and adding value to their business. In addition, IT usage provides benefits at several levels of businesses, government and society (Bin-Abbas & Bakry, 2014). IT is an important asset for supporting organizational processes. It is crucial for growth and innovation and for consolidation of fusions and acquisitions. However, according to Albertin and Albertin (2008), consistently determining the benefits of IT presents some challenges for managers due to the particularities of IT management. The problem involving IT theory and practice is how to maximize its potential (Wilkin & Chenhall, 2010). According to Nfuka and Rusu (2011), an understanding and familiarity with IT is essential to developing the alignment between business strategy and IT. In IT governance (ITG), CIOs must manage not only activities that are directly related to IT but also the connections between IT and organizational strategy (Schwarz & Hirschheim, 2003). According to Weill and Ross (2004), ITG is related to the definition of criteria for the management and assessment of IT investments. It can be considered an extension of Corporate Governance (CG), and because IT expenses have surpassed more than half of the capital of large companies (according to the US Department of Commerce (Economics & Statistics Administration, 2003)), managers are deeply concerned (Jordan & Musson, 2004). According to Gheorghe et al. (2009) and De Haes and Grembergen (2008), ITG is defined as procedures and policies established to guarantee that an organizationâ&#x20AC;&#x2122;s IT portfolio supports their objectives and strategies. To Butler and Butler (2010), and Lin, Chou and Wang (2011), ITG is mandatory for organizations due to the significant risks associated with ubiquitous business IT. ITG is a high priority for many organizations and high-level IT Governance models are being created (De Haes & Grembergen, 2008). To Simonsson, Johnson, and Ekstedt (2010), it is possible to identify a relationship between ITG performance and business performance, although there are no quantitative studies to prove its existence. In a study conducted by Weill and Ross (2005), companies that effectively govern IT can achieve 20% higher profits than other companies operating with similar strategies. Companies that effectively govern IT also have outcomes greater than their own capital and market capitalization growth. In this context, IT Governance helps to better control their projects. There is a link between high-level IT Governance performance results and high financial performance (Lunardi et al., 2014). Effective ITG aligns IT investments with business priorities, determines who makes the decisions about IT, and attributes the responsibility for the results. For financial institutions, it is particularly important to mitigate business risks and add value through IT usage (Gheorghe et al., 2009) in order to avoid fraud or guarantee forms to identify customers. According to Gheorghe et al. (2009), such demands have had an impact on ITG in financial institutions. This includes credit achievement, which now is measured by considering an organizationâ&#x20AC;&#x2122;s capacity to protect its assets, most of which are intangible. ITG is directly related to information security requirements and implementation of control policies to

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

399

guarantee that managers are truly aware of the company’s situation and are able to report it to the market, investors and other interested parties (stakeholders), thus reflecting their risk scenario, particularly in operations (Peck, 2006). There are several studies portraying ITG performance, such as the ones by Weill and Ross (2004, 2005), Jairak and Praneetpolgrang (2013), Albertin and Albertin (2008), Arruda and Silva Filho (2006) and Lunardi et al. (2014), among others. “However, having developed a high-level IT governance model does not imply that governance is actually working in the organization” (De Haes & Grembergen, 2008). According to Jong et al. (2010), Ko and Fink (2010), Krey et al. (2011), and Wilkin and Chenhall (2010), IT Governance is still a new and not fully developed field. It is necessary to develop additional work on theoretical and practical perspectives. In addition, one part of the sector that has benefited the most from IT development: bank automation. It is expected that bank automation’s high investments in IT follows a mature management system, i.e., a system that has implemented IT Governance. Through the adaptation of research conducted in 2002 by Weill and Ross (2004), this study is aimed at answering the following question: “What is the performance of IT governance in financial institutions operating in Brazil?” The main goal of this study is to measure the performance of ITG in financial institutions in Brazil, thereby making a future benchmark for worldwide results possible. To explore this goal, the following themes are conceptually approached: Corporate Governance (section 2), IT Governance (section 3), and Financial Institutions and their informatics process in Brazil (section 4). In addition, this paper presents the study’s methodology (section 5) and a discussion of the results (section 6).

CORPORATE GOVERNANCE

To understand IT Governance, it is necessary to first comprehend the concept of Corporate Governance (CG), the system by which organizations are directed and controlled. CG first appeared in the mid-1990s and was aimed at overcoming the conflict among agencies resulting from the separation of property and entrepreneurial management (IBGC, 2009) In this an unreliable and biased scenario, CG is intended to defend the adoption of key guidelines, such as transparency, income statements, equity, and corporative responsibility, to align management and stockholder interests and to avoid the abuse of authority, frauds, and strategic mistakes (IBGC, 2009). Corporate Governance is the system by which companies are directed, monitored and stimulated, and it involves relationships among owners, boards of directors, and control institutions. Governance is a mechanism for monitoring the actions, policies and decisions of corporations and involves the alignment of stakeholder interests (OECD, 2004). Corporate Governance best practices convert principles into recommended objectives and align interests to preserve and enhance the value of the organization, thereby making it easier to access capital and increase longevity (IBGC, 2009). The level of CG can be considered a parameter of the financial market. According to the research conducted by McKinsey and presented by Weill & Ross (2004), professional investors are ready to pay more to invest in companies with high governance standards, varying from 13% in North America to 25% in Asia. Large corporations attribute the same weight to CG as

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

400 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

they do to financial indicators when making investment decisions (Weill & Ross, 2004). According to the Brazilian Institution of Corporate Governance (IBCG, 2009), the World Bank and the International Monetary Fund “consider the adoption of best practices in corporate governance as part of the recovery of world markets weakened by the crisis of their respective capital markets.”

INFORMATION TECHNOLOGY GOVERNANCE

In general, CG approaches the relationship of six major types of assets in a corporation (human resources, financial, physical, intellectual property, information, and relationship). Due to low morale in the 1990s, entrepreneurs adapted to behavior change and required higher control, transparency and predictability, thereby influencing new generations of management tools. A succession of events made IT Governance essential in business (Mansur, 2007): the “2k bug”, which made clear the lack of knowledge about IT assets and how they impacted organizations; the “internet bubble”1, which showed budget concerns; and cases of fraudulent information in the financial sector, which showed the inefficiency of the existing rules and resulted in the creation of the Sarbanes-Oxley Act (SOX)2 (Wilkin & Chenhall, 2010). ITG is presented in a series of papers (Albertin and Albertin (2008), Schwarz and Hirschheim (2003), Van Grembergen (2004), Weill and Broadbent (2003), and Wilkin and Chenhall (2010)) as always being connected to decisions regarding the use and management of IT to overcome the information problems mentioned above. IT governance has been used to describe the policies, structures and processes of management that involve IT functions as a way to obtain a return on IT investments (Wilkin & Chenhall, 2010). According to Weill and Ross (2004), IT Governance is a process by which organizations align their IT actions with their performance goals and assess the consequences and results of these actions. Table 1 summarizes the major goals and advantages of effective ITG:

According to Peron (2009), a phenomenon was observed between 1995 and 2001, when dot-com companies were highly estimated in the stock exchange, stimulated by the promise of huge profits from this newly created virtual world. 2 The Sarbanes-Oxley Act strictly regulated the corporate world, establishing a new logic for the principles of CG and presenting itself as a way to renew best practices of legal conformity, responsibility for accounts, transparency and justice (IBGC, 2007).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

401

Table 1. Goals and advantages of IT governance. Goals of IT Governance:

IT Governance enables companies:



To facilitate investments.



To measure and audit the execution and quality of services



To streamline IT operations and/or IT services



To make feasible the follow-up of internal and external contracts



To improve the level of quality of IT services



To establish and maintain good relationships with clients and suppliers

To define conditions for the effective performance of management based on consolidated quality concepts.

decisions

about



To maximize the use of resources.



To optimize costs.



To manage risks (to identify, analyze, and mitigate them)



To establish and maintain conformity with rules and regulations



To promote integration Business and IT



To generate value for the company.

between

Advantages of IT Governance: 

Alignment of IT strategy with business fields;



Better quality and capacity for new models of business or adjustments in the current models



Maintenance of business risks under control;



Mediation and ongoing improvement of IT control;



Better transparency of IT activities

Source: Adapted from Tapajós (2008)

ITG is comprehensive and focused on transforming and directing IT resources towards current and future business requirements, both internal and external (Van Grembergen, Haes, & Guldentops, 2004; Wilkin & Chenhall, 2010). Weill and Ross (2004) summarized the key decisions about ITG in five domains (Table 2). The decisions regarding IT principles clarify the strategic role of IT business, establishing the guidelines for other decisions. Architecture decisions change IT principles into requirements for integration and standardization, achieving all of the technical specifications necessary to promote the required skills. Infrastructure decisions generate the necessary IT skills, while decisions involving the necessity of application use these skills. Finally, investment decisions mobilize and prioritize resources to transform system principles (Weill & Ross, 2004, 2005).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

402 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

Table 2. Major decisions about IT governance. IT principles: High-level decisions about the strategic role of IT in the business.

IT architecture:

IT infrastructure:

Integrated set of technical choices to guide the organization in satisfying business needs.

Centrally coordinated, shared IT services that provide the foundation for the enterpriseâ&#x20AC;&#x2122;s IT capability

Prioritization and investments Determine how much and where to invest in IT.

Business application needs: Business requirements for purchased or internally developed IT applications. Source: Weill and Ross (2004, p. 29)

Table 2 also depicts how IT is strongly related to business processes and strategic planning and, as highlighted by Weill and Ross (2004), how the IT field is not solely responsible for the effective use of data and IT. The responsibility has to be shared with company leaders. In general, ITG is related to two main factors: IT must add value to the business in terms of strategic alignment, and the risks of IT must be minimized, which requires the management of company resources. Both factors need to be supported by indicators and measures to guarantee that the intended results are achieved (Van Grembergen, Haes, & Guldentops, 2004; Wilkin & Chenhall, 2010). Value delivery and risk management are considered dimensions of result or governance purpose. Strategic alignment, resource management and performance measurement are understood as generator domains. When these three domains are well managed, they become feasible for the other two result domains (ITGI, 2003). Although many CEOs agree with the importance and necessity of IT Governance, its implementation is still considered a challenge in most corporations (Lunardi et al., 2007). Studies show that it is necessary to have a holistic approach to the knowledge involving ITG, such as resources, staff, and innovation. Because ITG is complex and dynamic, it consists of a series of interdependent subsystems that deliver a more powerful whole (Van Grembergen, Haes, & Guldentops, 2004). The answer to effective IT Governance implementation lies in a set of practices that refer to the structures, processes and forms of relationships (mechanisms of integration), as shown in Figure 1 (Van Grembergen, Haes, & Guldentops 2004).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

403

Processes

Structures

IT governance

Relationships Figure 1. IT Governance mechanisms Source: Adapted from Van Grembergen, Haes, & Guldentops, 2004, p. 5

Structure arrangements consist in organizing the units of a company and determining the ones responsible for IT decision-making, such as the creation of committees and strategic positions in IT (Van Grembergen, 2004). This dimension of the process focuses on the implementation of IT management techniques and the fulfillment of pre-established procedures such as IT strategies and policies. These practices provide acceptable measures for IT operations, such as clarity and transparency for governance. Implementation can be supported by using the Balanced Scorecard, COBIT (Control Objectives for Information and related Technology) (COBIT 5, 2013), and ITIL (Information Technology Infrastructure Library) (ITIL, 2013), among others (Bowen, Cheung, & Rohde, 2007). The integration mechanisms used determine the relationships and practices that make internal or external agents more involved with IT actions and consequences, thus promoting a better understanding and alignment among mechanisms (Lunardi et al., 2007). In addition to being more accepted by stakeholders, this set of models provides a more unified language and larger share of commitments. Stakeholders are better able to guide investments and monitor performance. This includes financial market analysts, a stakeholder group that is increasingly monitoring every data of IT “investment-performance” to provide better guidance to potential investors (ITGI, 2008). Therefore, these implementations are guided not only by the company’s internal characteristics but also by market requirements. From acts such as SOX to certifications such as ISO 20000, companies need to be regulated so that they can ensure their reliability in the market, among other factors. Failures in IT initiatives can lead to up to a 2% fall in the price of company stocks, according to a September 2009 report from a North American researcher (Young, 2009).

FINANCIAL INSTITUTIONS AND INFORMATION PROCESS

According to the Brazilian Bank Federation (FEBRABAN, 2010a), financial institutions are defined as follows:

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

404 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

Legal or private entities whose major or secondary activity is the collection, intermediation or investment of financial funds of their own or of third parties, in national or foreign currency, and the custody of property value of third parties. The Brazilian Bank Association also noted that in 2008, multiple and commercial banks were responsible for more than 80% of all assets in the National Financial System. As such, most of the information is concentrated in the bank sector rather than in other financial institutions (ABBC, 2008). In Brazil, bank automation follows three major points: the implementation of automation systems for administrative control, management support, and customer service. The concentration of financial capital, the fierce competition among banks, the different service offerings in the market, and the accounting standardization implemented by the Central Bank all enabled the first automation experiences in the late 1960s (Dalmazo, 2005). In the 1960s, the first Data Processing Centers (DPC) and accounting standardization were created. Made up of mainframes, the DPC stored all data from agencies’ daily financial operations (Blass, 1993). The same group started to operate investments, financing, brokerage, property finance, security and leasing (Dalmazo, 2005). In the 1970s, sub-centers of data input were implemented that filtered the information and sent it back to the DPC. After a short time, all data were automatically transferred to the processing centers. The first management decision support systems then appeared (Blass, 1993). In the 1980s, the use of magnetic cards began. The use of self-services also grew, including the use of automatic teller machines that were connected to banks, independent of place and time, and the so-called online banks, which offered customer service through automatic systems, completely changing the tellers’ operation routine. Thus, Multiple Banks3 were created. The advance of the technological standard was based on intensive, flexible and computer technologies in information and communication. As a result, a myriad of foreign banks came to Brazil (Dalmazo, 2005). In the 1990s, online systems were used internally by banks, decreasing or even eliminating the manual labor of some daily processes in the agencies (Blass, 1993). Since 1993, the number of self-service withdrawals and deposits in checking and savings accounts has been higher than those completed by agencies (Blass, 1993). The home banking system and the “pocket bank” were also created in the 1990s to make remote purchases and payments (Blass, 1993). After the Brazilian Central Bank’s reform in 2001 and 2002, the focus was shifted to risk management. The Brazilian Payment System enabled the transfer of resources, processing and payment settling to individuals, companies, the government, the Central Bank, and financial institutions. The Transfer of Reserves System introduced Brazil to a group of countries in which inter-bank transfer of funds can be performed in real time, unconditionally and irrevocably. The Transfer of Reserves System also guarantees security and reliability, helping with the reduction of Risco Brasil (country risk) (DEBAN, 2009).

A multiple bank is a private or public financial institution that performs lending, borrowing and accessory transactions by means of commercial, investment and/or development portfolios, property finance, commercial, credit, finance and investment leases (Banco Central do Brasil [BCB], 2007)

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

405

In 2004, bank IT management was still focused on technological aspects and was rarely integrated into the business management model. IT management was made up only of investments in hardware and software (Cordenonsi, 2004). Considering the strength of globalization and the growth of acquisitions and fusions, Brazilian banks are inclined to have more effective resource management and IT services. The quality of financial services is improved through the development and implementation of internal processes and controls to reduce the operational risk and by meeting the requirements of the market, government, account holders, and shareholders (Canton, 2008; Cordenonsi, 2004). Since 2008, this quality has advanced more than any other sector in terms of maturity in management by Corporate Governance, aside from leading IT Governance implementation (Canton, 2008). The COBIT is widely used by areas of IT and internal auditing in Brazilian financial institutions because the Central Bank follows this model for technical reference in the supervision of bank institution IT. Another market requirement, with external function restrictions, is that these financial institutions operate according to a regulation imposed by the government. The consequences of their actions have implications in their sector as well as political and social implications (Dalmazo, 2005; Toledo, 1993). In accordance with FEBRABAN (2010b), in 2008 the banking sectorâ&#x20AC;&#x2122;s IT budget surpassed R$16 billion, presenting an annual growth of 9%. The investments (recorded in the fixed banks) grew 12%, while current expenses increased 7%. It is interesting to note that in 2008, these investments represented 40% of all IT expenses, with amounts dedicated to the acquisition of cutting edge technological equipment and the development of new solutions, while the investments in telecommunications have become smaller and smaller each year (FEBRABAN, 2010b) Finally, it is possible to imagine that the â&#x20AC;&#x153;bank of the futureâ&#x20AC;? will be firmly supported by services coming from mobile platforms and the evolution of these technologies, as well as by telecommunications networks that will ensure a better data flow. Applying the client profile to each integration will be the key to applying technology to services that add intelligence to the most different processes. For example, there are more people using cell phones than there are account holders (CPqD, 2010).

METHODOLOGY

This study is considered a transversal descriptive survey. According to Malhotra (2001), this type of research is formal and structured and is based on quantitative data. The quantitative method is characterized by the use of quantification in both data gathering modalities and their treatment using statistics techniques (Richardson, 1999). Our data collection method was a survey, a procedure of collecting primary data from individuals. We used an electronic survey of self-managed questionnaires, and the survey was conducted without the physical presence of a researcher. The data collection instrument was based on a questionnaire developed and evaluated by Weill and Ross (2004, 2005) to assess IT Governance performance. In addition to the questions proposed by Weill and Ross, the questionnaire has two more parts. Table 3 summarizes the structure of the survey questionnaire, which has a total of 18 questions:

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

406 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

Table 3 - Set of variables of the survey instrument Group

Surveying goal

Source

Number of questions/statements

Organization classification

To trace the organization profile

Instituto Brasileiro de Geografia e Estatística [IBGE] (2004)

Functional classification of the interviewed

To trace the profile of the respondent professional

Arruda and Silva Filho (2006)

Relevance and success of the IT activities

To obtain information about ITG performance

Weill and Ross (2004)

Our study initially required that information about the respondents be used in the descriptive analysis, forming a data set about the profile of the respondents who participated in it, on behalf of their respective financial organizations. As mentioned above, the group of questions about relevance and success of IT activities is based on the instrument developed and validated by Weill and Ross (2004, 2005), which evaluates IT Governance performance.

5.1 SURVEY MODEL The questions related to relevance and success of IT activities are divided into two blocks: relevance and influence. Questions in the “relevance block” evaluate the importance of specific IT results. This block considers the importance of IT. The questions in the “influence block” evaluate which IT governance measure contributes to the achievement of these results (Weill & Ross, 2005). Each answer to the questions in the relevance block and the influence block corresponds to a numeric value and block (i.e., Q1 and Q2, respectively), as shown in Table 4: Table 4 - Numeric value of the questions in the Relevance and Influence blocks “Relevance block” (Q1)

“Influence block” (Q2)

No relevance = 1

No Success = 1

Little relevance = 2

Little success = 2

Medium relevance = 3

Regular = 3

Relevant = 4

Well succeeded = 4

Very relevant = 5

Very well succeeded = 5

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

407

The answers that composevariable Q1 attribute value to the answers of variable Q2. The sum of the average of the four objectives is multiplied by 100 (maximum total to achieve) and divided by the companies’ maximum score. The same relevance was considered, as was the fact that a company was very successful in every aspect, using a mathematical Equation 1: (Σn=a 4 (Relevant of result {Q1}) * Influence of IT governance {Q2})) *100 ( Σn=1 to 4 (5 (Relevant of result)) Equation 1 - Calculation formula for performance of IT governance Source: Weill and Ross (2004)

In a nutshell, the calculation expressed in Equation 1 can be simplified by using the spreadsheet presented in Table 5. Table 5 - Spreadsheet for the calculation of IT governance performance How important are the following outcomes of your IT governance?

How does your IT governance influence these outcomes?

Total

Cost-effective use of IT

Effective use of IT for the company’s growth

Effective use of IT for asset utilization

Effective use of IT for business flexibility

Total of Relevance

Total

Source: Weill and Ross (2004)

The formula to calculate IT governance performance can be expressed by the reduced form according to Equation 2: (Total / Total of Relevance) X 20 Equation 2 – Reduced formula for calculation of performance of IT governance Source: Weill and Ross (2005)

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

408 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

PRESENTATION AND ANALYSIS OF RESULTS

The questionnaire was available online from April 12, 2010, to June 1, 2010. The web address (URL) was sent to the respondents by electronic mail, and invitation e-mails were sent to representative members from the Brazilian Society for Worldwide Interbank Financial Telecommunication (SWIFT) (FEBRABAN, 2010c). Using the FEBRABAN website, 96 highranking employees from financial institutions operating in Brazil were contacted. A second database of employees from the financial institutions that are part of the housing finance system was also used, and 25 of them were licensed and made available on the Urban Development Ministry’s website. Other invitations were made based on random contacts and the academic environment with the participation of former students and the current staff of financial institutions.

6.1. SAMPLE PROFILE CHARACTERIZATION In total, there were 105 answers, out of which 60 were complete and 45 were incomplete. The data were exported to an electronic spreadsheet in which more advanced calculations and tables were made. To facilitate the collection of data, the country/region was not specified. In addition, due to the content of some questions, it was necessary to keep respondents anonymous. The financial institutions in the sample are classified as follows: 56% are quasigovernment companies (both public and private capital), 71.5% are multiple banks with a commercial portfolio, and 86.5% are institutions that employ more than 100 people and have an annual gross income higher than R$2.400.000,00 (approximately US$1.351.351,35 in 18th January 2012.) The respondents’ profile classification is as follows: 60% are part of the operational staff and not in a leadership position, 63% have been employed in the same organization for more than 5 years, 65% work in the IT sector, and 74% interact with the information system on a daily basis, whereas 51% spend more than 4 hours a day using the information system.

6.2. BLOCKS OF QUESTIONS ANALYSIS The “relevance block” evaluates how the four objectives are considered; that is, it evaluates the importance of specific results (Weill & Ross, 2004). As follows, each question of the block is analyzed. The element considered to be the most important was the use of IT for the growth of the organization; second, the use of IT for business flexibility; third, to maximize the cost/benefit relationship of the decisions; and fourth, the efficient use of assets. The “relevance block” reflects the performance companies deliver when using IT in the four questions. It evaluates in what measure ITG contributes to these results (Weill & Ross, 2004). The results obtained for each question demonstrated that the sample considers the most successful company to be the one that uses IT to reach organizational growth. Ranked second is the use of IT to maximize the cost/benefit relationship and the use of IT for business flexibility. Ranked last is the use of IT for better asset use.

6.3. ITG PERFORMANCE RESULTS To measure IT governance performance according to the formula given by Weill and Ross (2004) (equation 1), it is necessary to follow some mathematic steps, as shown in Section 5.1. Considering that every organization does not give the same importance to the results, the answers to the first question (Table 6) attribute some weight to the answers of the second

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

409

questions (second part of Table 6). The scores considered by the weight of the four questions are added and divided by the maximum score achieved by the company (Weill & Ross, 2004).

Answer / Relevance

Table 6 - Q1 and Q2 Calculation Score 1 2 3 4 5

Q1 Average

Cost/Benefit 3 0 0 22 33 256 4,413793103

Growth 2 2 1 16 37 258 4,448275862

Asset 2 1 3 25 27 248 4,275862069

Flexibility 2 1 4 16 35 255 4,396551724

Answer/ Inlfuence

58 Score 1 2 3 4 5

Q2 Average

Cost/Benefit Growth Asset Flexibility 2 1 3 3 2 3 2 1 6 6 8 7 32 29 31 31 16 19 14 16 232 236 225 230 4,00000000 4,068965517 3,879310345 3,965517241

Considering Equation 1, the multiplication of the score addition (Q1 and Q2) = 69.77705, and the addition of Q1 = 87.67241. Thus, we have 69.77705*100/87.67241 = 79.58838. The value of IT governance performance was approximately 79.6.

6.4. DISCUSSION OF RESULTS Because there are four objectives, the maximum score is 100 and the minimum score is 20, it is possible to consider 60 to be an average score that indicates regular/medium/good performance. According to the sample of the sector of the Brazilian financial institutions, it can be observed that a performance between maximum and regular is calculated to be approximately 80 points. In other words, according to how much the sector prioritizes the use of IT in its activities, it has apparently achieved a satisfactory and effective result. Regarding the aspects that an organization considers as the most important, such as the use of IT for growth and increased business flexibility, there should be more efforts to achieve results. As demonstrated by Canton (2008), in 2010, it was expected that the banks were more concerned about meeting legal requirements, such as BasilĂŠia II. This finding would mean that ITG performance has found its place; however, there are still blanks to be filled to achieve a maximum score, such as looking for a better use of IT in the cost/benefit relationship of activities and use of assets, that would generate higher business value and improve resource management. In the survey developed by Weill and Ross (2004), who considered 256 Gartner EXP4 worldwide member companies, the average score was 69, and one-third of the companies 4

Gartner, Executive Program of Gartner Inc., Company of survey and global consultancy about IT for companies. Founded in 1979, located in Stamford, Connecticut, EUA, present in 80 countries, with more than 4300 employees and partners.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

410 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

scored above 74. In Brazil, Gama and Martinello (2006) conducted a study and concluded that only 10% of the 30 companies surveyed presented an ITG level lower than 70. In achieving approximately 79.6 points, the Brazilian financial institutions show that there is a bias towards a positive evolution in the performance of IT governance; however, it is possible to question how fast it is really happening and whether the relevant elements will also undergo changes over time.

CONCLUSIONS

The present study measured the IT governance performance of financial institutions operating in Brazil using a sample of professionals in the area. The samples used consist mostly of operational bank workers who have worked for their company for more than 5 years and are part of the IT team. The relevance given to the use of IT for decision making, flexibility, growth, and resource management were mostly considered to be very high. Nevertheless, IT’s influence on final performance, which generates company results in the above mentioned perspectives, did not reach its maximum performance, a factor that is possible to identify by a score of approximately 79.6. It is important to highlight that the IT Governance concept is still new and that its practice and theoretic basis are still being developed. In this respect, the present paper is relevant because its results can help understand how this administrative practice is being developed in Brazil. Theoretically, topics on IT governance are infrequently presented by schools and scholars, and a large part of the material available is informal. Regarding the empirical aspect, although the intention is to analyze the sector as a whole, the low level of diversity in the financial institutions and academic background of the respondents must be considered. Another limitation of the study is the low index of answers, which is most likely related to the respondents’ pre-disposition towards the topic or the lack of influence on entities related to the theme, which could be assisted. In this manner, it was not possible to use a probable sample and attribute results for only the sample elements. The main contribution of this research is the application of a significant method that provides a benchmark for an important area of economy. The results reflect some type of reliability for IT governance in financial institutions. The research can be considered a significant point for attracting investments to the sector. Scientifically, it can be a basis for further studies or comparisons over time considering the impacts of IT on society. From the present study’s results and limitations, some points appeared that could be interesting for future studies, including a deeper investigation of the themes identified in the literature review that have yet to be adequately explored, such as the issue of IT implementation and its barriers. Another possibility could be to identify the relationship between performance in each IT governance domain and global performance. The financial variables needed to prove influences on organizations’ accounting sector, which could also be studied. For future works, another suggestion is an exploratory work that conducts an in-depth interview with employees of a major financial institution to further explain the answers obtained in this survey. Finally, this study could be redone in different countries, sectors, or estates, aimed at making comparisons and thus clarifying the real scenario of IT Governance implementation and performance.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

411

REFERENCES

ABBC (2008) Associação Brasileira de Bancos. Relatório anual. Recuperado em 25 maio 2010, de http://www.abbc.org.br/abbc/web/arquivos/relatorio_2008_final.pdf Afzali, P., Azmayandeh, E., Nassiri, R., Shabgahi, G. L. (2010). Effective governance through simultaneous use of COBIT and val IT. Proceedings of the International Conference on Education and Management Technology, 2, 46-50. Albertin, A. L., Albertin, R. M. (2008). Benefícios do uso de tecnologia de informação para o desempenho empresarial. Revista de Administração Pública, 42(2), 275-302. Arruda, P. A. F., Silva Filho, J. B. (2006). Governança de tecnologia da informação para micro e pequenas: Estudo de caso na cidade de Fortaleza. Anais do XXVI ENEGEP, Fortaleza, CE. Ayat, M., Masrom, M., Sahibuddin, S. (2011). Issues in implementing it governance in small and medium enterprises. Proceedings of the 2nd International Conference on Intelligent Systems, Modelling and Simulation (ISMS), Phnom Penh, Cambodia. BCB (2007). Banco Central do Brasil. Banco Múltiplo: Definição. Recuperado em 31 maio 2010, de http://www.bacen.gov.br. Bin-Abbas, H.; Bakry, S. H. (2014). Assessment of IT governance in organizations: A simple integrated approach, Computers in Human Behavior, Volume 32, March 2014, Pages 261-267, ISSN 0747-5632, http://dx.doi.org/10.1016/j.chb.2013.12.019. Blass, L. M. S. (1993). Automação bancária, práticas e representação. São Paulo em Perspectiva, 7(4), 81-89. Bowen, P. L., Cheung, M. D., & Rohde, F. H. (2007). Enhancing IT governance practices: A model and case study of an organization's efforts. International Journal of Accounting Information System, 8(3), 191-221. Butler, R., & Butler, M. (2010). Beyond King III: Assigning accountability for IT governance in South African enterprises. South African Journal of Business Management, 41(3), 33. Canton, E. P. (2008). Governança de TI nas instituições financeiras no Brasil: Uma avaliação de tendências. Dissertação (mestrado), Centro Estadual de Educação Tecnológica Paula Souza, São Paulo, SP. COBIT 5 (2013): A business framework for the governance and management of enterprise IT. Information System Audit and Control Association. <http://www.isaca.org/cobit> Cordenonsi, J. L. (2004). Um modelo de administração da tecnologia da informação. São Paulo: Atlas. CPqD. (2010). Banco do Futuro: Tecnologia para excelência bancária. Recuperado em 24 maio 2010, de http://www.cpqd.com.br/ofertas-relacionadas/247-temas-estruturantes/4731-banco-dofuturo.html Dalmazo, M. S. (2005). A evolução do atendimento bancário: A importância do marketing dentro das instituições financeiras. Trabalho de conclusão de curso, Centro Universitário Nove de Julho, São Paulo, SP. DEBAN (2009). Banco Central do Brasil, Departamento de Operações Bancárias e de Sistema de Pagamentos. Sistema de pagamentos brasileiro. Disponível em: http://www.bacen.gov.br. Brasília, DF

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

412 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

De Haes, S.; Grembergen, W. V., (2008) Analysing the Relationship between IT Governance and Business/IT Alignment Maturity, Hawaii International Conference on System Sciences, Proceedings of the 41st Annual , vol., no., pp.428,428, 7-10 Jan. 2008 doi: 10.1109/HICSS.2008.66 Economics & Statistics Administration. (2003). Digital economy 2003. Washington, DC: US Department of Commerce. FEBRABAN (2010a) Federação Brasileira de Bancos. Portal de informações do setor. Recuperado em 25 maio 2010, de http://www.febraban.org.br FEBRABAN (2010b) Federação Brasileira de Bancos. O setor bancário em números. Recuperado em 24 maio 2010, de http://www.febraban.org.br/acervo.asp?id_pagina=85&id_paginaDe=78&id_texto= FEBRABAN (2010c) Federação Brasileira de Bancos. SWIFT. Recuperado em 24 maio 2010, de http://www.febraban.org.br/Febraban.asp?modulo=Servi%E7os&id_pagina=110 Gama, F. A., & Martinello, M. (2006). Governança de tecnologia da informação: Um estudo em empresas brasileiras. Trabalho apresentado no 4o Simpósio FUCAPE, Vitória, ES. Gheorghe, M., Nastase, P., Boldeanu, D., & Ofelia, A. (2009). IT Governance in Romania: A case study. Global Economy Journal, 9(1), 1-13. IBGE (2004). Instituto Brasileiro de Geografia e Estatística. CNAE – Classificação Nacional de Atividades Econômicas. Recuperado em 10 novembro 2009, de http://www.cnae.ibge.gov.br/ IBGC (2007). Instituto Brasileiro de Governança Corporativa. Código de melhores práticas de governança corporativa (3a ed.). São Paulo: IBGC. IBGC (2009) Instituto Brasileiro de Governança Corporativa. Governança corporativa. Recuperado em 26 novembro 2009, de http://www.ibgc.org.br/Secao.aspx?CodSecao=17 ITGI (2003) IT Governance Institute. Board briefing on IT Governance (2nd ed.). Rolling Meadows, IL: ITGI. ITGI (2008) IT Governance Institute. Unlocking value: An executive primer on the critical role of IT governance. Rolling Meadows, IL: ITGI. ITIL (2013). Information Technology Infrastructure Library, the British Office of Government Commerce. <www.itil.org> Jairak, K.; Praneetpolgrang, P.; (2013) Applying IT governance balanced scorecard and importance-performance analysis for providing IT governance strategy in university, Information Management & Computer Security, Vol. 21, 4, pp.228 - 249 Jong, F., Van Hillegersberg, J., Van Eck, P., Van Der Kolk, F., & Jorissen, R. (2010). Governance of offshore IT outsourcing at shell global functions IT-BAM development and application of a governance framework to improve outsourcing relationships. In: W. van der Aalst, J. Mylopoulos, M. Rosemann, M. J. Shaw, C. Szyperski (Series Eds.), I. Oshri, & J. Kotlarsky (Vol. Eds.), Lecture notes in business information processing: Vol. 55. Global sourcing of information technology and business processes (pp. 119-150). Berlin: Springer. Jordan, E., & Musson, D. (2004). Corporate Governance and IT Governance: Exploring the board's perspective. Proceedings of the ACIS, Sydney, Australia. Ko, D., & Fink, D. (2010). Information technology governance: An evaluation of a theorypractice gap. Corporate Governance, 10(5), 662-674.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

Performance Measurement Of Information Technology Governance in Brazilian Financial Institutions

413

Krey, M., Keller, T., Harriehausen, B., & Knoll, M. (2011). Towards a classification of information technology governance frameworks for the development of a IT GRC healthcare framework. IEEE International Workshop on Consumer eHealth Platforms, Services and Applications, 1, 34-38. Lin, F., Chou, S., & Wang, W. K. (2011). IS practitioners views on core factors of effective IT governance for Taiwan SMEs. International Journal of Technology Management, 54(2), 256269. Lunardi, G. L.; Becker, J. L.; Maçada, A. C. G.; Dolci, P. C. (2014) The impact of adopting IT governance on financial performance. A impirical analysis among Brazilian firms. International Journal of Accounting Information Systems, Volume 15, Issue 1, March, Pages 66–81 Lunardi, G. L., Dolci, P. C., Becker, J. L., & Gastaud, A. C. (2007). Governança de TI no Brasil: Uma análise dos mecanismos mais difundidos entre as empresas nacionais. Trabalho apresentado no Simpósio de Excelência em Gestão e Tecnologia, Rio Grande do Sul, RS. Malhotra, N. K. (2001). Pesquisa de marketing: Uma orientação aplicada. Porto Alegre, RS: Bookman. Mansur, R. (2007). Governança de TI: Metodologia, frameworks e melhores práticas. Rio de Janeiro, RJ: Brasport. Nfuka, E. N., & Rusu, L. (2011). The effect of critical success factors on IT governance performance". Industrial Management & Data Systems, 111(9), 1418-1448. OECD (2004), Principles of Corporate Governance. OECD. Available on <http://www.oecd.org/corporate/oecdprinciplesofcorporategovernance.htm>. Retrieved 201406-24. Peck, P. (2006, 17 janeiro). Quais são os impactos legais da Sarbanes-Oxley no Brasil? CIO. Recuperado em 24 maio 2009, de http://cio.uol.com.br/gestao/2006/01/18/idgnoticia.20060118.5258602125/ Richardson, R. J. (1999). Pesquisa social: Método e técnicas. São Paulo, SP: Atlas. Schwarz, A., & Hirschheim, R. (2003). An extended platform logic perspective of IT governance: Managing perceptions and activities of IT. Journal of Strategic Information System, 12(2), 129-166. Simonsson, M., Johnson, P., & Ekstedt, M. (2010). The effect of IT Governance maturity on IT Governance performance. Information Systems Management, 27(1), 10-24. Tapajós, U. (2008). Governança de TI com ITIL® (Vol. 3). São Paulo, SP: CompanyWeb. Toledo, L. G. (1993). Marketing bancário: Análise, planejamento, processo decisório. São Paulo, SP: Atlas. Van Grembergen, W. (2004). Strategies for information technology governance. Hershey, PA: Idea Group. Van Grembergen, W., De Haes, S., & Guldentops, E. (2004). Structures, processes and relational mechanisms for IT Governance. In W. Van Grembergen (Ed.), Strategies for information technology governance (pp. 1-36). Hershey, PA: Idea Group. Weill, P., & Broadbent, M. (2003). Creating effective IT Governance: A Gartner EXP Premier Survey Report. Stamford, CT: Gartner. Weill, P., & Ross, J. (2004). IT Governance: How top performers manage IT decision rights for superior results. Boston, MA: Harvard Business School Press.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

414 Barbosa, S. C. B., Rodello, I. A., Padua, S. I. D. de

Weill, P., & Ross, J. (2005). A matrixed approach to designing IT Governance. MIT Sloan Management Review, 46(2), 26-34. Wilkin, C., & Chenhall, R. (2010). A review of IT Governance: A taxonomy to inform accounting information system. Journal of Information System, 24(2), 107-146. Young, R. (2009, 25 agosto). Pesquisa: Falha em TI reduz até 2% o valor de ações negociadas em bolsa. e8Consulting: CIO. Recuperado em 20 novembro de 2009, de http://cio.uol.com.br/opiniao/2009/08/25/pesquisa-falha-em-ti-reduz-ate-2-o-valor-de-acoesnegociadas-em-bolsa/

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.397-414

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 437-460 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200012

CONTINUOUS DATA AUDITING AS AN AUTOMATION TOOL FOR BUSINESS CONTROL AUDITORIA CONTÍNUA DE DADOS COMO INSTRUMENTO DE AUTOMAÇÃO DO CONTROLE EMPRESARIAL Washington Lopes da Silva Tecban S/A, São Paulo, SP, Brazil Jorge Rady de Almeida Jr. University of São Paulo, São Paulo, SP, Brazil _____________________________________________________________________ ABSTRACT Business activities and business processes in the corporate world are technology-dependent. Such activities and processes drive the development of new audit techniques to investigate possible systemic failures that could affect the internal controls of companies. In this way, the need for automation of auditing tests led to the development of the concept and implementation of projects for continuous data auditing in the enterprise environment. This article evaluates the major critical issues for the construction of continuous data auditing, considering the use of information technology and computer assisted audit techniques. Four basic premises were formulated based on the theoretical foundation. From them eight propositions and eleven critical aspects were derived, which were placed in a proof of concept, through a multiple case study in the Brazilian business scenario. The main findings of this research were the critical aspects for the construction of continuous data auditing, originated in theoretical foundation and corroborated by the results of a multiple case study, which will serve as drivers for the initialization of continuous auditing projects as well as for their restructuring. Keywords: Continuous Audit; Continuous Monitoring; Real-Time; Database. _____________________________________________________________________________________________

Manuscript first received/Recebido em: 20/04/2013 Manuscript accepted/Aprovado em: 05/02/2014 Address for correspondence Washington Lopes da Silva, Possui graduação em Ciências Contábeis pela Faculdade de Ciências Econômicas de São Paulo (1995), mestrado em Engenharia Elétrica Área de Concentração Engenharia da Computação, pela Universidade Presbiteriana Mackenzie (2000), doutorado em Engenharia Elétrica Área de Concentração em Sistemas Digitais, pela Escola Politécnica da Universidade de São Paulo (2012). Superintendente de Auditoria Interna da TecBan S/A. Professor contratado pela FIPECAFI - Fundação Instituto de Pesquisas Atuariais, Contábeis EMail: washington.lopes2@usp.br Jorge Rady de Almeida Jr., Possui mestrado em Engenharia Elétrica pela Escola Politécnica da Universidade de São Paulo (1990) e doutorado também em Engenharia Elétrica pela Escola Politécnica da Universidade de São Paulo (1995). É Professor Associado do Departamento de Engenharia de Computação e Sistemas Digitais (PCS) da Escola Politécnica da Universidade de São Paulo desde 2003. Suas áreas de pesquisa são o Desenvolvimento e Avaliação de Sistemas Críticos, com ênfase nos aspectos de tempo real e a área de Engenharia de Software, com ênfase em Bancos de Dados, Data Warehouse e Data Mining, e segurança em Sistemas de Informação. E-Mail: jorge.almeida@poli.usp.br Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

438 Silva, W.L. ,Júnior, J.R. de A.

1. INTRODUÇÃO A constante evolução e dependência tecnológica das atividades e dos processos de negócios no mundo corporativo, o aumento da oferta de novos produtos, serviços e canais de venda, além do crescimento das transações eletrônicas a partir da globalização, deram origem à necessidade do desenvolvimento tecnológico da área de Auditoria para apurar possíveis falhas sistêmicas que pudessem afetar os controles internos das companhias. Adicionalmente, a rigidez das exigências regulamentares, como a Lei americana Sarbanes-Oxley e os princípios de Governança Corporativa que propõem um nível elevado de transparência e um ambiente de controle interno organizado e bem gerenciado, incrementaram a importância da Auditoria, seja ela interna ou externa, nas organizações. A necessidade de segurança instantânea e constante sobre a eficiência do gerenciamento de riscos e o ambiente de controles internos é crítica. As organizações estão expostas a erros significativos, fraudes e ineficiências que podem gerar perdas financeiras e a elevação da exposição aos riscos. De acordo com Church; Mcmillan e Schneider (2001), os auditores têm um papel relevante na avaliação dos processos de negócios, principalmente naqueles voltados para garantir a integridade das demonstrações financeiras, e devem alertar a administração sobre qualquer indício de fraude. A área de Auditoria vem sofrendo diversas transformações. Pode-se resumir que a auditoria trabalhava com o passado, ou seja, relatava a história, corrigia e evitava a recorrência. No presente e futuro, a Auditoria deverá avaliar cenários, estratégias, com foco nos resultados dos negócios, ou seja, estará apta a prevenir a ocorrência de falhas e problemas. 1.1 Justificativa Os estudos existentes sobre Auditoria Contínua demonstram que o ideal para o auditor seria trabalhar com dados amostrais mais próximos da realidade das exceções a serem analisadas. Quais práticas de auditoria tradicionais sobreviverão na era da economia digital, onde as organizações, a cada dia que passa, ficam mais dependentes de sistemas e dados computacionais? Vale também ressaltar que o ambiente de governança corporativa, no qual a auditoria se insere, relaciona as políticas, os princípios e as práticas geridas pelas organizações, que devem criar um conjunto eficiente de mecanismos para assegurar que o comportamento dos seus gestores, bem como os controles internos, estejam sempre na mesma linha que seus acionistas. Assim, uma das formas das organizações atingirem a confiança do seu público interno e externo, através do monitoramento dos controles de seus processos críticos de negócio, pode ocorrer por meio da auditoria contínua de dados. Segundo Vasarhelyi, Alles, e Kogan (2002), as pressões do cenário corporativo apontam para que o processo de auditoria passe a ser efetuado de forma contínua. Este JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

439

tipo de auditoria fornece aos auditores um conjunto de ferramentas que poderá transformar o processo de auditoria em um processo de monitoração contínua dos processos de negócio de uma organização. 1.2 Objetivo Este trabalho pretende colaborar com o aprofundamento do conhecimento das técnicas e modelos utilizados de auditoria contínua de dados, integrando o uso da tecnologia da informação e seu relacionamento com a auditoria interna e governança corporativa, principalmente no que diz respeito à disponibilização de dados computacionais para o efetivo trabalho de gestão organizacional no contexto empresarial brasileiro. De acordo com Vasarhelyi e Alles (2008), a Auditoria Contínua não representa apenas um estudo acadêmico em que se utilizam softwares que estão sendo desenvolvidos e oferecidos por empresas privadas. Apesar dos desafios enfrentados, a Auditoria Contínua será o futuro da auditoria empresarial, tendo em conta a tecnologia moderna, como nos sistemas de ERP e através de leis como a Lei Sarbanes-Oxley. Desta forma, este trabalho questiona: “Quais os principais aspectos críticos para a construção da auditoria contínua de dados, considerando o uso da tecnologia da informação e das técnicas de auditoria com auxílio do computador (TAAC)?” Trata-se de uma pesquisa documental, de cunho qualitativo. Primeiramente foi realizada uma varredura na literatura sobre auditoria, tecnologia da informação e seus desdobramentos para a gestão organizacional. Além da profunda pesquisa bibliográfica sobre auditoria, tecnologia da informação e banco de dados, foi proposto um estudo de casos múltiplos, considerando os principais aspectos críticos relacionados à auditoria e ao ambiente tecnológico que suportam os processos críticos de negócios das organizações brasileiras usuárias de processos de auditoria contínua. Foram coletados dados primários através de pesquisas bibliográficas e, na sequência, dados secundários com os responsáveis das áreas de Auditoria Interna e Tecnologia da Informação. Por fim, foi feita uma prova de conceito, contextualizando os aspectos críticos que devem ser considerados pela área de Auditoria Interna para a implantação da auditoria contínua. 1.3 Contribuição A contribuição deste trabalho está inserida nos projetos de implantação de auditoria contínua. Primeiramente foram identificados os principais aspectos críticos a serem considerados durante a construção da Auditoria Contínua. Posteriormente, foram selecionadas empresas que já implantaram ou estão em processo de implantação de rotinas de auditoria contínua.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

440 Silva, W.L. ,Júnior, J.R. de A.

Os casos pesquisados refletem o cenário atual das empresas brasileiras de grande porte, com um nível alto de complexidade organizacional de seus processos de negócios e que, por uma necessidade de aumentar a abordagem da auditoria de forma mais tempestiva, tiveram como diretriz a automação de seus testes de auditoria, com a utilização da auditoria contínua de dados. Os aspectos críticos identificados e corroborados pelas empresas pesquisadas, contribuem para as diretrizes de implantação da auditoria contínua, prevendo situações que poderão gerar barreiras durante a construção do projeto. Com a identificação de tais aspectos, as organizações interessadas poderão endereçar ações preventivas que contribuirão para o andamento do projeto e para o atingimento dos resultados esperados. O conceito atual de auditoria contínua está relacionado com a disponibilidade de dados mais próximos de um evento, se possível em tempo real, com capacidade de serem processados e correlacionados em um ambiente computadorizado seguro, que traga informações eletrônicas fidedignas, que serão tratadas pelos auditores e gestores da organização. 2.METODOLOGIA DA PESQUISA A Figura 1 representa o modelo teórico sobre a questão principal da pesquisa, objeto central desta pesquisa. Nesta Figura, o foco concentra-se em destacar os aspectos críticos para a construção da auditoria contínua de dados, levando em conta a Tecnologia da Informação e as Técnicas de Auditoria com Auxílio do Computador. Tendo tal foco e sua respectiva fundamentação teórica, formularam-se quatro premissas básicas, cujo objetivo é demonstrar como devem se relacionar as áreas de Auditoria Interna, Tecnologia da Informação, processos de negócios, sistemas, banco de dados e ambientes computacionais, para se chegar à construção da Auditoria Contínua de Dados. Além disso, foram desenvolvidas oito proposições que direcionam a pesquisa de campo para os aspectos críticos a serem examinados, de acordo com o foco de estudo. Tais aspectos críticos foram validados após a aplicação da pesquisa de campo que foi desenvolvida por meio de estudo de casos múltiplos.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

Proposição 1 Processos de negócios Premissa 1

automatizados, requer monitoração automática de

441

Aspecto 1 Capacidade de desenvolver e manter um nível adequado de automação dos processos de negócios.

seus controles e riscos Para a implantação da auditoria contínua de dados é necessário que a organização possua um nível adequado de automação de seus processos de negócios e controles associados.

associados. Aspecto 2

Proposição 2 Processos de Negócios são suportados por controles automáticos que se relacionam entre os diversos sistemas e

Aspecto 3

seus respectivos módulos

Capacidade que os sistemas, banco de dados e controles automáticos possuem de se inter-relacionarem.

funcionais.

Proposição 3 Premissa 2

O papel do engenheiro do

A área de Tecnologia da Informação, a área de Auditoria Interna e demais áreas das organizações devem se relacionar de forma a atingir os objetivos esperados para o atendimento das necessidades dos processos de auditoria contínua de dados.

interação com os auditores

conhecimento deve ser de

Questão Principal Quais os principais aspectos críticos para a construção da Auditoria Contínua de Dados, considerando o uso da Tecnologia da Informação e das Técnicas de Auditoria com Auxílio do Computador?

internos e analistas de

Aspecto 5 Proposição 4 Buscar sinergia com as

Habilidade da área de TI em atender de forma consistente e sistemática as necessidades de automação da área de Auditoria Interna.

diversas áreas da organização e aproveitar as estruturas

Proposição 5 Sistemas apropriados e seguros que geram dados íntegros e com qualidade para

Os ambientes computacionais disponibilizados pela área de TI para a área de Auditoria Interna devem ser confiáveis, de forma a fornecer dados íntegros para a automação dos processos de auditoria.

Aspecto 4 Habilidade da área de Auditoria Interna em identificar de forma consistente e sistemática os testes e indicadores de auditoria que são objetos de automação.

sistemas da área de TI.

tecnológicas existentes.

Premissa 3

Capacidade que os sistemas implantados possuem para gerar informações para o monitoramento contínuo.

a auditoria contínua de dados.

Proposição 6 Os sistemas que gerarão os dados para a auditoria contínua deverão ser avaliados periodicamente pelos auditores

Aspecto 6 Capacidade de se identificar possíveis sinergias entre as áreas da organização ganhando eficiência e reduzindo possíveis retrabalhos.

Aspecto 7 Capacidade do ambiente de TI absorver de forma rápida e eficiente o desenvolvimento das rotinas de Auditoria Contínua.

Aspecto 8 Capacidade do ambiente de TI de manter as rotinas de auditoria contínua em um ambiente estável e confiável, de forma a atender as necessidades dos usuários.

de TI.

Aspecto 9 Proposição 7

Assegurar que os sistemas e o ambiente de TI que suportam as rotinas de auditoria contínua geram dados confiáveis.

O planejamento para a auditoria contínua de dados

Premissa 4 Os processos de automação de auditoria para alcançar a auditoria contínua de dados, devem ser escolhidos de acordo com o grau de risco dos processos de negócios chaves, avaliados pela organização.

deve seguir método

Aspecto 10

semelhante ao usado para o

Capacidade da Auditoria Interna de cobrir os principais riscos dos processos críticos de negócios, por meio da auditoria contínua de dados.

planejamento auditoria.

Proposição 8 Visão de risco única de forma a concentrar esforços na implantação de processos automáticos que mitigarão os

Aspecto 11 Habilidade da Auditoria Interna em compartilhar sua visão de riscos com as demais áreas da organização.

riscos.

FUNDAMENTAÇÃO TEÓRICA

PESQUISA DE CAMPO

Figura 1 – Premissas e proposições associadas deste trabalho

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

442 Silva, W.L. ,Júnior, J.R. de A.

2.1 Premissas, Proposições, Aspectos Críticos E Questões Associadas 2.1.1 Premissa 1 Para Yeh e Shen (2010), um sistema de auditoria contínua deve ter como critério um gerenciamento dos controles e do ambiente computacional e sistemas que suportam a automação dos processos. De acordo com Flowerday, S; Blundell, A. W; Solms, R. Von. (2006), na era da contabilidade e comunicação em tempo real, as práticas atuais de auditoria, quando efetivas, provêm resultados de auditoria após a ocorrência de uma fraude ou erro. A asseguração contínua pode auxiliar na prevenção de erros intencionais ou não intencionais. Isto pode ser atingido através da auditoria contínua, a qual é suportada grandemente pelo uso da tecnologia da informação. Segundo BOVEE et al. (2005), vale destacar que existem estudos de modelos de automação, baseados em agentes inteligentes, responsáveis por converter e consolidar demonstrações financeiras, incluindo balanços patrimoniais e fluxos de caixas em XBRL, como o já existente protótipo denominado FRAANK – Financial Reporting and Auditing Agent with Net Knowledge. Uma técnica de auditoria eficaz para o protótipo proposto seria o uso da auditoria contínua. Para Alles, M.; Brennan, G.; Kogan, A.; Vasarhelyi, M. A. (2005), a experiência com a evolução das novas tecnologias e processos de negócios sugerem que a auditoria contínua inicialmente será usada para fazer mais do que automação de processos existentes de auditoria e, além disso, ela ganhará vantagens das funcionalidades dos sistemas ERP. O segundo estágio da evolução se dá quando os processos de auditoria são redesenhados (reengenharia de processos) para explorar toda a capacidade tecnológica disponível nos sistemas. De acordo com Vasarhelyi, Alles, e Kogan (2004), a auditoria contínua começa ganhar mais espaço à medida que as organizações vão ganhando maior automação em seus processos de negócios e, como consequência, as exigências para monitoramento dos riscos de negócios também crescem. Deste cenário infere-se a primeira premissa deste estudo com suas respectivas proposições, aspectos críticos e questões associadas: Premissa 1 (PRE-1): para a implantação da auditoria contínua de dados é necessário que a organização possua um nível adequado de automação de seus processos de negócios e controles associados. Proposição 1 (PRO-1): as organizações dependentes de TI possuem seus processos de negócios automatizados, o que torna essencial uma monitoração automática dos dados advindos de seus controles e riscos associados.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

443

Aspecto Crítico 1: Capacidade de desenvolver e manter um nível adequado de automação dos processos de negócios. Questão 1: Como a área de TI está estruturada para atender às necessidades de automação dos processos de negócios da organização? Aspecto Crítico 2: Capacidade que os sistemas implantados possuem para gerar informações para o monitoramento contínuo dos processos de negócios. Questão 2: Os sistemas que suportam os processos de negócios possuem controles automáticos e procedimentos de governança de TI suficientes para garantir o monitoramento dos riscos associados aos processos, ou ainda há a necessidade de intervenções de controles manuais? Proposição 2 (PRO-2):os processos de negócios das organizações dependentes de TI são suportados por controles automáticos que se relacionam entre os diversos sistemas e seus respectivos módulos funcionais. Aspecto Crítico 3: Capacidade que os sistemas, banco de dados e controles automáticos possuem de se inter-relacionarem. Questão 3: Os sistemas em produção, bem como seus respectivos bancos de dados se inter-relacionam, de forma a manter um fluxo de informação controlado e consistente com os objetivos dos processos de negócios?

2.1.2 Premissa 2 Rezaee, Z.; Sharbatoghlie, A.; Elam, R.; Mcmickle, L.P. (2002) comentam que um alto grau de automação dos processos envolve módulos de auditoria integrados com a aplicação, que será constantemente monitorada. Um grau menor de automação envolve a habilidade para automaticamente capturar, transformar e carregar dados, mas isto requer o envolvimento de um auditor capacitado para isoladamente processar relatórios de exceção e detectar falhas nos controles. De acordo com Coderre (2005), o auditor deverá ter acesso às bases de dados da organização para construir a auditoria contínua. Para isso, também é necessário verificar como poderá ser dado o acesso a estes dados, de forma a não prejudicar o desempenho do ambiente computacional em questão e também não concorrer com os demais processamentos das diversas áreas da organização. Chou, C. L.; Du, T.; Lai, V. S. (2007) definem (EAM – Embedded Audit Modules ou Módulos de auditoria integrados) como sendo pequenos módulos que são construídos dentro de um sistema maior para capturar informações e gerar reportes de situações fora do padrão esperado. A construção destes módulos de auditoria depende JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

444 Silva, W.L. ,Júnior, J.R. de A.

da participação do auditor no projeto de implementação, juntamente com os desenvolvedores da área de TI. Chou et al. (2007) também comentam que o uso de data marts requer habilidade por parte do auditor gerenciar e manusear o volume de dados dos data marts de auditoria. Li, S.-H.; Huang, S.-M.; Lin, Y.-C. G. (2007) comentam que os auditores precisam ter entendimento sobre as aplicações sistêmicas e como elas suportam os processos de negócios. Menciona também que os auditores devem solicitar, quando necessário o suporte do DBA (Administrador do Banco de Dados ou Database Administrator) para ter acesso de consulta aos dados que serão foco da auditoria. Para Hunton e Rose (2010), os Sistemas de Suporte a Decisão ou DSS –Decision Support System – podem auxiliar os auditores nas análises dos indicadores gerados através dos diversos processos das áreas de negócios da organização, bem como no próprio planejamento dos trabalhos dos auditores, auxiliando como um direcionador de análises para os riscos mais relevantes. Outro fator a ser considerado é como se chegar aos objetivos esperados da auditoria contínua. De acordo com Vasarhelyi e Halper (1991), os modelos estatísticos e heurísticos do sistema CPAS(Continuous Process Auditing System) foram desenvolvidos considerando técnicas de inteligência artificial e engenharia do conhecimento. Desta forma, é possível inferir a segunda premissa deste estudo com suas respectivas proposições, aspectos críticos e questões associadas: Premissa 2 (PRE-2): a área de Tecnologia da Informação (TI), a área de Auditoria Interna e demais áreas das organizações devem se relacionar de forma a atingir os objetivos esperados para o atendimento das necessidades dos processos de auditoria contínua de dados. Proposição 3 (PRO-3): o papel do engenheiro do conhecimento agregado à área de Auditoria Interna deve ser de interação com os auditores internos e analistas de sistemas da área de TI, formando um elo entre os interessados na automação dos processos de auditoria e quem desenvolve os sistemas para processar as rotinas de auditoria contínua. Aspecto Crítico 4: Habilidade da área de Auditoria Interna em identificar de forma consistente e sistemática os testes e indicadores de auditoria que são objetos de automação. Questão 4: A área de Auditoria Interna possui auditores especialistas, com conhecimento específico em desenvolvimento de sistemas e que identificam as necessidades de automação dos testes e indicadores de auditoria? Aspecto Crítico 5: Habilidade da área de TI em atender de forma consistente e sistemática as necessidades de automação da área de Auditoria Interna.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

445

Questão 5: A área de TI possui analistas de sistemas dedicados para atender às necessidades da área de Auditoria Interna? Proposição 4 (PRO-4): a área de Auditoria Interna deve buscar sinergia com as diversas áreas da organização, verificando se existe a possibilidade de se aproveitar as estruturas tecnológicas existentes com seus respectivos dados associados, como exemplo, os ambientes de Data Warehouse já em utilização pelas diversas áreas da organização. Aspecto Crítico 6: Capacidade de se identificar possíveis sinergias entre as áreas da organização ganhando eficiência e reduzindo possíveis retrabalhos. Questão 6: Existem outras áreas na organização que possuem processos automatizados de monitoramento contínuo de dados? Aspecto Crítico 7: Capacidade do ambiente de TI absorver de forma rápida e eficiente o desenvolvimento das rotinas de Auditoria Contínua. Questão 7: A área de TI possui um ambiente de banco de dados flexível com capacidade de suportar a Auditoria Contínua, como exemplo um ambiente de Data Warehouse?

2.1.3 Premissa 3 Coderre (2005) cita que para se construir a auditoria contínua é necessário obter confiança dos sistemas que gerarão dados para as rotinas de auditoria contínua. Para Yeh e Shen (2010), um sistema de auditoria contínua confiável consiste em dados confiáveis, íntegros e disponíveis, dentro de um ambiente de controle interno gerenciado. Segundo Flowerday et al. (2006), quando o auditor avalia os reportes produzidos pela auditoria contínua, ele verifica a confidencialidade, integridade e disponibilidade dos dados e como estes são suportados pelos sistemas de controles internos. Flowerday et al. (2006) também comentam que um dos problemas que afeta as soluções de auditoria contínua é a variedade de formatos de dados e registros, incluindo os sistemas legados que são cruciais para a criação do sistema de auditoria contínua. Para isso é necessário que haja uma avaliação e padronização destes dados para que ocorram erros de processamento e registros duplicados. Desta forma, é possível inferir a terceira premissa deste estudo, com suas respectivas proposições, aspectos críticos e questões associadas: Premissa 3 (PRE-3): os ambientes computacionais disponibilizados pela área de TI para a área de Auditoria Interna devem ser confiáveis de forma a fornecer dados íntegros para a automação dos processos de auditoria. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

446 Silva, W.L. ,Júnior, J.R. de A.

Proposição 5 (PRO-5): sistemas apropriados e seguros que geram dados íntegros e com qualidade são essenciais no processo de construção da auditoria contínua de dados. Aspecto Crítico 8: Capacidade do ambiente de TI de manter as rotinas de auditoria contínua em um ambiente estável e confiável, de forma a atender as necessidades dos usuários. Questão 8: Os dados que suportam as rotinas de auditoria contínua são processados em um ambiente de TI com controles gerais confiáveis, garantindo a disponibilidade, desempenho, segurança, capacidade de ser redimensionado, flexibilidade e continuidade operacional? Proposição 6 (PRO-6): a validação dos sistemas que gerarão os dados para a auditoria contínua deverá ser avaliada periodicamente pelos auditores de TI, assegurando as informações geradas pelas diversas rotinas automatizadas de auditoria. Aspecto Crítico 9: Assegurar que os sistemas e o ambiente de TI que suportam as rotinas de auditoria contínua gerem dados confiáveis. Questão 9: A área de Auditoria Interna mantém programas de trabalhos aplicáveis pelos auditores de TI dentro do ciclo de auditoria para abordar os aspectos críticos de governança do ambiente TI e seus respectivos controles gerais? 2.1.4 Premissa 4 De acordo com Alles et al. (2005), os riscos a serem avaliados pelas organizações no processo de auditoria contínua devem partir de um modelo “framework” reconhecido pelas melhores práticas, como por exemplo, o modelo COSO – Committee of Sponsoring Organizations of the Treadway Commission reconhecido mundialmente. Para Donnell e Schultz (2003), os auditores podem suportar sua metodologia de avaliação de riscos de negócios através de softwares focados nos processos de negócios (BPF – Business Process Focused). O sistema apresenta a cadeia de valores da organização, priorizada por seus riscos de negócios ou através de softwares focados no ciclo de uma transação (TCF – Transaction Cycle Focused), onde o sistema classifica as transações de acordo com sua relevância contábil. Para Koch (1984), as técnicas de auditoria com auxílio do computador são similares à execução de uma auditoria: as transações são selecionadas, inseridas no sistema, processadas pela aplicação, os resultados são comparados e as exceções são notificadas. Cohen, Krishnamoorthy e Wright (2000) comentam que para a escolha dos testes e procedimentos analíticos de auditoria, os auditores devem considerar informações financeiras e não financeiras, tais como condições econômicas, mudanças tecnológicas, condições mercadológicas nas quais a organização está inserida etc. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

447

Para Glover, Jiambalvo e Kennedy (2000), os auditores devem preliminarmente planejar seus testes, executá-los posteriormente, mas devem revisar e redirecionar seu plano a qualquer momento, considerando inclusive informações da administração que podem afetar a exposição aos riscos previamente avaliados. Rezaee et al. (2002) comentam que os testes de controle devem ser contínuos. Isto permite ao auditor expressar uma opinião mais confiável a respeito do sistema de controles internos. Para isso, é citado que a confiança nos controles internos é definida durante a fase de planejamento da auditoria. De acordo com LI et al. (2007), o auditor deve ter conhecimento dos processos de negócio da organização para estabelecer um programa efetivo de auditoria. Ainda segundo LI et al. (2007), para determinar a auditoria contínua, o auditor deve estabelecer os objetivos do trabalho, por meio de assertivas e as cita:  Existência: Verificar se as transações registradas nos sistemas existem de fato;  Completude: Verificar se todas as transações existentes foram registradas no sistema;  Acuracidade: Verificar se as transações foram registradas com seus valores corretos;  Classificação: Verificar se as transações foram classificadas corretamente em suas devidas contas contábeis;  Tempestividade: Verificar se as transações foram registradas na data correta;  Valorização: Verificar se as transações foram corretamente calculadas e valorizadas. Desta forma, é possível inferir a quarta premissa deste estudo, com suas respectivas proposições, aspectos críticos e questões associadas: Premissa 4 (PRE-4): os processos de automação de auditoria, para alcançar a auditoria contínua de dados, devem ser escolhidos de acordo com o grau de risco dos processos de negócios-chave, avaliados pela organização. Proposição 7 (PRO-7): o planejamento para a auditoria contínua de dados deve seguir método semelhante ao usado para o planejamento cíclico da auditoria, de forma a priorizar as entidades auditáveis de alto risco. Aspecto Crítico 10: Capacidade da Auditoria Interna de cobrir os principais riscos dos processos críticos de negócios por meio da auditoria contínua de dados. Questão 10: Como a área de Auditoria Interna define e prioriza o desenvolvimento e execução das rotinas de auditoria contínua de dados? Proposição 8 (PRO-8): a visão de risco da organização deve ser única, de forma a concentrar os esforços na implantação de processos automáticos que mitigarão os riscos associados aos controles.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

448 Silva, W.L. ,Júnior, J.R. de A.

Aspecto Crítico 11: Habilidade da Auditoria Interna em compartilhar sua visão de riscos com as demais áreas da organização. Questão 11: Como a Auditoria Interna compartilha da visão de riscos dos demais departamentos da organização para definir a cobertura da auditoria contínua?

3. ESTUDO DE CASOS MÚLTIPLOS De acordo com Alles, Tostes, Vasarhelyi e Riccio (2006), a auditoria contínua pode ser definida como uma transformação da auditoria interna e externa, através da aplicação da moderna tecnologia da informação. Além disso, está sendo adotada por diversas empresas no mundo, como Siemens, HCA, The Royal Canadian Mounted Police, BIPOP Bank etc. Como apresentado nos diversos simpósios e palestras sobre o tema, como o 25th WCARS - World Continuous Auditing and Reporting Systems Symposium -, em conjunto com o 9° CONTECSI - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação, em 2012 e o 22th WCARS, em conjunto com o 8° CONTECSI em 2011, a auditoria contínua está em sua fase inicial de implantação pelas empresas brasileiras. Por isso, o foco da pesquisa é em empresas que possuem processos de auditoria contínua consolidados ou que tenham projetos de auditoria contínua em fase adiantada, próximos de atingir seus objetivos. O estudo de casos múltiplos foi realizado por meio de uma pesquisa de campo, através de uma entrevista semiestruturada que permitiu maior liberdade do pesquisador, com os principais gestores das organizações pesquisadas. O modelo de estudo de casos múltiplos, ilustrado na Figura 2, se utilizou do construto proposto, ilustrado na Figura 1, embasado na fundamentação teórica, na elaboração prévia das premissas, proposições, aspectos críticos e questionamentos associados. Atualmente no contexto nacional existem poucas organizações de diversos setores que já implementaram auditoria contínua, o destaque fica para as grandes instituições financeiras, que somam uma quantidade de 6 grandes conglomerados financeiros. Dessa forma, a seleção das empresas foi direcionada para aquelas que possuem seus processos de auditoria contínua já consolidados, de forma a testar todos os aspectos críticos levantados neste trabalho. Para não concentrar a amostra, foram selecionadas três empresas do setor financeiro e duas de outros setores que já possuem seus sistemas de auditoria contínua em produção. As respostas aos questionamentos permitiram explorar e descrever os casos pesquisados e direcionar as respostas para a questão principal da pesquisa, procurando manter a objetividade e a isenção na análise dos documentos. Para conseguir maior segurança na prova de conceito, foi feita uma análise cruzada das informações obtidas junto às áreas de Auditoria Interna e Tecnologia da Informação das organizações pesquisadas, de forma a correlacionar os resultados da JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

449

pesquisa e chegar às conclusões sobre os principais aspectos críticos para a construção da auditoria contínua de dados, considerando o uso da tecnologia da informação e das técnicas de auditoria com auxílio do computador. Tal análise forneceu afirmações ou negações conclusivas de cada resposta obtida de cada um dos questionamentos elaborados. A compilação dos resultados foi conduzida de forma sistemática e documentada, conforme a convergência entre as respostas obtidas das empresas. Primeiramente foi feita uma descrição de cada resposta de cada questionamento e, na sequência, foram compiladas as afirmações ou negações associadas de cada aspecto crítico. Cada aspecto crítico foi analisado em um Quadro específico. De acordo com Yin (2010), a quarta e última modalidade de relatório escrito aplica-se apenas a estudo de casos múltiplos. Nessa situação, não pode haver capítulos ou seções separados destinados a casos individuais. Em seu lugar, o relatório inteiro consiste em uma análise cruzada, mesmo que seja puramente descritivo ou que lide com tópicos explanatórios. Finalmente, após fazer as análises isoladas de cada aspecto crítico, foi feita uma composição de todos os aspectos críticos abordados na pesquisa de campo, utilizando-se do modelo descrito, onde se verificou a aderência conjunta dos resultados obtidos por meio da análise cruzada, chegando-se às considerações finais sobre a questão principal deste trabalho. Cabe ressaltar que os modelos dos quadros para composição dos resultados do estudo de casos múltiplos foram baseados no método utilizado por Biancolino (2010).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

450 Silva, W.L. ,Júnior, J.R. de A.

Questão Principal

Fundamentação Teórica

Premissas

Trabalhos Futuros

Proposições

Aspectos Críticos

Questões de Pesquisa

Estudos de Casos Múltiplos (5 Casos)

Seleção dos Casos

Pesquisa em Campo

Análise Cruzada

Conclusões da Pesquisa

Figura2 – Modelo do Estudo de Casos Múltiplos

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

451

3.1 Resultados Preliminares 3.1.1 Aderência Aos Aspectos Críticos Associados À PRE-1, PRO-1 E PRO-2 Com base na fundamentação teórica e na pesquisa de campo, desenvolvida por meio de estudo de casos múltiplos, os principais aspectos críticos para a construção da auditoria contínua de dados, associados ao nível adequado de automação dos processos de negócios e seus respectivos controles das organizações, podem ser resumidos pelas informações demonstradas no Quadro 1: Código do Aspecto Crítico

AC1

AC2

PRO1

AC3 PRO-2

Aspectos abordados Capacidade de desenvolver e manter um nível adequado de automação dos processos de negócios. Capacidade que os sistemas implantados possuem para gerar informações para o monitoramento contínuo dos processos de negócios. Grau de Aderência da Proposição 1

Aderência Verificada

100%

90%

95%

Capacidade que os sistemas, banco de dados e controles automáticos, possuem de se inter-relacionarem.

100%

Grau de Aderência da Proposição 2

100%

Grau de Aderência Geral PRE-1

Aspectos Críticos Relacionados à Premissa 1

97%

Quadro 1 – Grau de aderência geral – Aspectos abordados relacionados à Premissa 1 De acordo com o Quadro 1, pode-se concluir que é extremamente relevante que as organizações, as quais possuem ou pretendem possuir processos de auditoria contínua, mantenham um nível adequado de automação de seus processos de negócios. Cabe ressaltar que, por meio das proposições de pesquisa e dos números apresentados no Quadro 1, pode-se concluir também que tais processos de negócios, quando automatizados, requerem monitoração automática de seus controles e riscos associados, os quais se inter-relacionam entre os diversos sistemas e seus respectivos módulos funcionais.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

452 Silva, W.L. ,Júnior, J.R. de A.

3.1.2aderência Aos Aspectos Críticos Associados À PRE-2, PRO-3 E PRO-4 Com base na fundamentação teórica e na pesquisa de campo, desenvolvida por meio de estudo de casos múltiplos, demonstramos abaixo, no quadro 2, os principais aspectos críticos para a construção da auditoria contínua de dados - associados ao relacionamento que as áreas de TI, Auditoria Interna e as demais áreas da organização deve ter, para atingir os objetivos esperados das necessidades dos processos de auditoria contínua de dados. Código do Aspecto Crítico

Aspectos abordados

Aderência Verificada

AC4

Habilidade da área de Auditoria Interna em identificar de forma consistente e sistemática os testes e indicadores de auditoria que são objetos de automação.

100%

AC5

Habilidade da área de TI em atender de forma consistente e sistemática as necessidades de automação da área de Auditoria Interna.

50%

Grau de Aderência da Proposição 3

75%

AC6

Capacidade de se identificar possíveis sinergias entre as áreas da organização ganhando eficiência e reduzindo possíveis retrabalhos.

20%

AC7

Capacidade do ambiente de TI absorver de forma rápida e eficiente o desenvolvimento das rotinas de Auditoria Contínua.

60%

PRO-4

Grau de Aderência da Proposição 4

40%

PRO-3

PRE-2

Grau de Aderência Geral Aspectos Críticos Relacionados à Premissa 2

58%

Quadro 2 – Grau de aderência geral – Aspectos abordados relacionados à Premissa 2 O Quadro 2 demonstra que é relevante as áreas de TI e Auditoria Interna, bem como as demais áreas da organização, se relacionarem para atingir os objetivos esperados da auditoria contínua. De certa forma, pode-se inferir que a importância dada a isto se deve ao fato de que todas as áreas da organização são responsáveis por gerenciar seus controles e riscos e a Auditoria Interna. Utilizando-se da auditoria contínua, a organização conseguirá contribuir para o atingimento deste objetivo.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

453

A proposição 3 ressalta que é muito relevante o papel do engenheiro do conhecimento ou o auditor especialista em auditoria e em TI para manter um nível adequado de interação entre os auditores internos e analistas de sistemas da área de TI. Já a proposição 4 ressalta sua pouca relevância para atingir os objetivos da auditoria contínua, quando se trata de identificar possíveis sinergias pela Auditoria Interna, entre as áreas das organizações que possuem processos automáticos de monitoramento contínuo. Disto pode-se inferir que as áreas de Auditoria Interna dão preferência a ter processos próprios de auditoria contínua, mantendo assim seu princípio de independência das demais áreas da organização. De qualquer forma, na proposição 4 é demonstrada a relevância, quando se trata, isoladamente, do aspecto crítico da capacidade do ambiente de TI absorver de forma rápida as rotinas de auditoria contínua. Durante a pesquisa pôde-se constatar que é fundamental que as áreas de TI disponibilizem um ambiente de TI funcional, capacitando a área de Auditoria Interna para suportar e expandir seus processos de auditoria contínua. Para isto, é imprescindível a sinergia e interação entre as áreas de TI e Auditoria Interna, a fim de suprir as necessidades da auditoria contínua, no que diz respeito à disponibilização de ambiente tecnológico. 3.1.3 ADERÊNCIA AOS ASPECTOS ASSOCIADOS À PRE-3, PRO-5 E PRO-6 Com base na fundamentação teórica e na pesquisa de campo, desenvolvida por meio de estudo de casos múltiplos, os principais aspectos críticos para a construção da auditoria contínua de dados, associados aos ambientes computacionais disponibilizados pela área de TI para a Auditoria Interna que devem ser confiáveis, de forma a fornecer dados íntegros para a automação dos processos de auditoria, podem ser resumidos pelas informações demonstradas no Quadro 3: Código do Aspecto Crítico

Aspectos abordados

Aderência Verificada

AC8

Capacidade do ambiente de TI de manter as rotinas de auditoria contínua em um ambiente estável e confiável, de forma a atender as necessidades dos usuários.

100%

PRO-5

Grau de Aderência da Proposição 5

100%

AC9

Assegurar que os sistemas e o ambiente de TI que suportam as rotinas de auditoria contínua geram dados confiáveis.

100%

PRO-6

Grau de Aderência da Proposição 6

100%

PRE-3

Grau de Aderência Geral Aspectos Críticos Relacionados à Premissa 3

100%

Quadro 3 – Grau de aderência geral – Aspectos abordados relacionados à Premissa 3

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

454 Silva, W.L. ,Júnior, J.R. de A.

De acordo com o Quadro 3, pode-se concluir que é extremamente relevante que as organizações, as quais possuem ou pretendem possuir processos de auditoria contínua, tenham ambientes computacionais confiáveis e que disponibilizem dados íntegros para a automação dos processos de auditoria. Da mesma forma, pode-se concluir, por meio das proposições, a extrema relevância desse ambiente computacional processar sistemas apropriados e seguros que gerem dados íntegros e com qualidade para a auditoria contínua. Para se assegurar desta proposição, as áreas de Auditoria Interna possuem uma equipe específica de auditores de TI que aplicam programas de trabalho para auditar os sistemas aplicativos, infraestrutura e processos de governança de TI de seus ambientes computacionais. 3.1.4 ADERÊNCIA AOS ASPECTOS ASSOCIADOS À PRE-4, PRO-7 E PRO-8 Com base na fundamentação teórica e na pesquisa de campo, desenvolvida por meio de estudo de casos múltiplos, os principais aspectos críticos para a construção da auditoria contínua de dados, associados à escolha dos processos de automação de auditoria, os quais devem ser selecionados de acordo com o grau de risco dos processos de negócios-chave, podem ser resumidos pelas informações demonstradas no Quadro 4: Código do Aspecto Crítico

AC10

PRO-7

Aspectos abordados

Capacidade da Auditoria Interna de cobrir os principais riscos dos processos críticos de negócios, por meio da auditoria contínua de dados.

Grau de Aderência da Proposição 7

Aderência Verificada

100%

AC11

Habilidade da Auditoria Interna em compartilhar sua visão de riscos com as demais áreas da organização.

80%

PRO-8

Grau de Aderência da Proposição 8

80%

PRE-4

Grau de Aderência Geral Aspectos Críticos Relacionados à Premissa 4

90%

Quadro 4 – Grau de aderência geral – Aspectos abordados relacionados à Premissa 4 De acordo com o Quadro 4, pode-se concluir que é muito relevante que as organizações, as quais possuem ou pretendem possuir processos de auditoria contínua, selecionem seus processos de automação de auditoria de acordo com o grau de risco dos JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

455

processos de negócios-chave. Para que isto aconteça, a pesquisa demonstrou que todas as áreas de Auditoria Interna utilizam um método para avaliar os riscos de todos os seus processos auditáveis, incluindo aqueles que serão alvo da auditoria contínua. Foi também constatado que é muito relevante às organizações possuírem uma visão de riscos única entre as suas diversas áreas, fazendo com que a Auditoria Interna concentre seus esforços na implantação de rotinas de auditoria contínua para os processos de negócios de maior risco, os quais podem expor a organização. Conforme verificado nas análises cruzadas e de aderências associadas às premissas de estudo, pode-se concluir que, com base na fundamentação teórica e do estudo de casos múltiplos, os principais aspectos críticos para a construção da auditoria contínua de dados, quando consolidados nas premissas que os originaram, obtiveram um percentual de 86% de aderência, como observado no Quadro 5.

Código da Premissa

PRE-1

PRE-2

PRE-3

PRE-4

Aderência Verificada

Premissas Para a implantação da auditoria contínua de dados é necessário que a organização possua um nível adequado de automação de seus processos de negócios e controles associados. A área de Tecnologia da Informação (TI), a área de Auditoria Interna e demais áreas das organizações devem se relacionar de forma a atingir os objetivos esperados para o atendimento das necessidades dos processos de auditoria contínua de dados. Os ambientes computacionais disponibilizados pela área de TI para a área de Auditoria Interna devem ser confiáveis, de forma a fornecer dados íntegros para a automação dos processos de auditoria. Os processos de automação de auditoria, para alcançar a auditoria contínua de dados, devem ser escolhidos de acordo com o grau de risco dos processos de negócios-chave, avaliados pela organização.

97%

58%

100%

90%

Grau de Aderência Geral Premissas derivadas da Fundamentação Teórica

86%

Quadro 5 – Grau de aderência geral – Premissas derivadas da Fundamentação Teórica JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

456 Silva, W.L. ,Júnior, J.R. de A.

4. CONCLUSÃO Partindo-se do método de apuração deste tipo de pesquisa proposto por Biancolino (2010), é importante frisar que os valores atribuídos às análises qualitativas foram inseridos na pesquisa apenas como referências quantitativas simples. Os números apresentados nesta pesquisa com relação às análises de aderência efetuadas têm por objetivo situar as respectivas aderências em uma escala de valores, de forma que os aspectos críticos para a construção da auditoria contínua, extremamente relevantes para os entrevistados, se aproximam do valor de 100%, ao mesmo tempo em que os irrelevantes, no mesmo cenário, se aproximam de zero. Cabe ressaltar que as análises de aderências efetuadas nesta pesquisa foram classificadas de acordo com a opinião dos entrevistados, sem que houvesse influência por parte do pesquisador. Desta forma, os aspectos críticos foram classificados de acordo com uma abordagem relativa, segundo o método de apuração aplicado. Respeitadas as limitações da pesquisa e o método aplicado, é possível concluir que os principais aspectos críticos para a construção da auditoria contínua de dados, considerando o uso da Tecnologia da Informação e das Técnicas de Auditoria com Auxílio do Computador, podem ser representados através de uma distribuição espacial associada à sua relevância, conforme demonstrado na Figura 3.

100 AC1

AC3

Extremamente Relevante

AC4

AC8

AC9

AC10

AC2

AC11 Muito Relevante

75 AC7

AC5

Relevante

Pouco Relevante

25 AC6 Irrelevante

Premissa 1

Premissa 2

Premissa 3

Premissa 4

Figura 3 – Graus de aderência dos aspectos críticos propostos A partir da Figura 3, juntamente com todas as demais análises realizadas nesta pesquisa, pode-se concluir que: JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

457

a) Observa-se que 8 aspectos críticos propostos nesta pesquisa encontram-se situados no patamar acima de 75% de aderência, enquanto que 2 aspectos situam-se entre 50% e 75% de aderência e apenas 1 abaixo de 25%. Conclui-se a partir deste fato que os aspectos críticos apresentados estão presentes no cotidiano das organizações que possuem seus processos de auditoria contínua de dados em produção ou em fase de implantação. b) Todos os aspectos críticos apresentados na Premissa 1 apresentaram-se acima de 75% de aderência. Sendo assim, pode-se considerar que a auditoria contínua de dados é totalmente dependente do nível adequado de automação dos processos de negócios das organizações, sem o qual não haveria condições de implantar rotinas de auditoria contínua. Disto pode-se inferir que as organizações de maior porte, com a grande maioria de seus processos de negócios dependentes de TI, possuem maiores condições de construir suas auditorias contínuas de dados. c) Os aspectos críticos relacionados à Premissa 2 não se apresentaram de forma homogênea no gráfico. Isto se deve ao fato de as áreas de Auditoria Interna manterem equipes especializadas para o funcionamento dos processos de auditoria contínua, de forma independente, fazendo inclusive o papel dos analistas de sistemas da área de TI na definição de aplicativos e infraestrutura tecnológica para processar as rotinas de auditoria contínua, extrapolando o papel do engenheiro do conhecimento para a área de Auditoria Interna. Este fato é evidente devido à falta de atendimento consistente por parte da área de TI para as necessidades de automação dos processos de Auditoria Interna, dificultando inclusive a obtenção de dados para o funcionamento das rotinas de auditoria contínua. Além disso, ficou claro que ainda falta maior sinergia entre as diversas áreas das organizações e a Auditoria Interna para se atingir maior eficiência na implantação dos processos de auditoria contínua de dados. Isto acontece pelo não aproveitamento da infraestrutura de TI, dos ambientes de banco de dados disponíveis e dos processos automáticos de monitoração já existentes nas organizações, os quais poderiam ser usados para a construção da auditoria contínua. d) A homogeneidade dos aspectos críticos das Premissas 3 e 4 se situarem em patamares elevados, se deve ao fato de que qualquer processo de auditoria contínua, quando colocado em produção, deve ser considerado confiável em todos os seus aspectos estruturais, desde sua escolha, de acordo com o grau de risco do objeto auditável, passando-se pela obtenção dos dados, processamento, chegando-se no resultado final. É importante ressaltar que a auditoria contínua de dados foi considerada pelas organizações como mais uma das técnicas de auditoria, a qual deve ser utilizada de acordo com a oportunidade percebida durante o processo de análise de riscos que define os processos auditáveis. e) Como conclusão final, afirma-se que os aspectos críticos para a construção da auditoria de dados, originários da fundamentação teórica e corroborados pelo resultado do estudo de casos múltiplos, servirão como direcionadores para o início de projetos de auditoria contínua, bem como para sua reestruturação. A pesquisa permitiu que, com novos estudos e a partir dos resultados apresentados, outras análises a respeito do tema, possam ser desenvolvidas.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

458 Silva, W.L. ,Júnior, J.R. de A.

4.1 TRABALHOS FUTUROS Através de uma análise detalhada dos assuntos desenvolvidos nesta pesquisa, sugerem-se temas que podem complementar o estudo sobre a auditoria contínua de dados: 1) Estudo detalhado de cada aspecto crítico pesquisado nesta pesquisa. 2) Pesquisar em detalhes como as organizações migram seus atuais processos de auditoria para a auditoria contínua de dados, considerando os aspectos críticos estudados. 3) Proposição de um método de construção da auditoria contínua de dados, partindo-se dos aspectos críticos pesquisados. 4) Estudo sobre os ambientes de bancos de dados e seu aproveitamento na Auditoria Interna, utilizando-se da auditoria contínua de dados. 5) Os efeitos da modernização da Auditoria Interna, utilizando-se da auditoria contínua de dados. 6) Os efeitos dos resultados da auditoria contínua de dados para a melhoria da governança corporativa e o aumento da sinergia entre as áreas das organizações. 7) Pesquisa aprofundada das diversas formas de obtenção de dados e métodos estatísticos para se chegar aos resultados da auditoria contínua. 8) O uso da auditoria contínua de dados e seus efeitos em sistemas ERPs. 9) O uso da auditoria contínua de dados como instrumento da prevenção a fraudes nas organizações. 10) Estudo sobre a formação do profissional de auditoria especialista em processos de auditoria contínua de dados.

REFERÊNCIAS BIBLIOGRÁFICAS Alles, M.; Brennan, G.; Kogan, A.; Vasarhelyi, M. A. (2005). Continuous monitoring of business process controls: a pilot implementation of a continuous auditing system at Siemens. International Journal of Accounting Information Systems, v.7, p.137161. Alles, M. G. A.; Tostes, F.; Vasarhelyi, M. A.; Riccio, E. L. (2006) Continuous Auditing: The USA Experience and Considerations for its Implementation in Brazil. Journal of Information Systems and Technology Management.v.3, n.2, p.211-214. Biancolino, C. A. (2010). Valor de uso do ERP e gestão contínua de pósimplementação: estudo de casos múltiplos no cenário brasileiro. Pesquisa (Doutorado) São Paulo: Universidade de São Paulo. JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

Auditoria Contínua de Dados como Instrumento de Automação do Controle Empresarial

459

Bovee, Matthew; Kogan, Alexander; Nelson, Kay; Srivastava, Rajendra P. (2005). Financial Reporting and Auditing Agent with Net Knowledge (FRAANK) and eXtensible Business Reporting Language (XBRL). Journal of Information Systems.v.19, n.1, p.19-41. Chou, C. L.; Du, T.; Lai, V. S. (2007).Continuous auditing with a multi-agent system. Decision Support Systems, v.42, n.4, p.2274-2292. Church, B. K.; Mcmillan, J. J.; Schneider, A. (2001).Factors Affecting Internal Auditors’ Consideration of Fraudulent Financial Reporting during Analytical Procedures. A Journal of Practice & Theory, v. 20, n.1. Coderre, D. (2005). Continuous Auditing: Implications for Assurance, Monitoring and Risk Assessment. GTAG#3 - Global Technology Audit Guide. USA. The IIA - The Institute of Internal Auditor. Donnell, E. O.; Schultz, J. J. (2003).The Influence of Business-Process-Focused Audit Support Software on Analytical Procedures Judgments. A Journal of Practice & Theory.v.22, n.2, p.265-279. Flowerday, S; Blundell, A. W; Solms, R. Von.(2006). Continuous auditing technologies and models: a discussion. Computers & Security, n.25, p. 325-331. Glover, S. M.; Jiambalvo, J.; e Kennedy, J. (2000).Analytical Procedures and AuditPlanning Decisions. A Journal of Practice & Theory.v.19, n.2. Hunton, J. E.; Rose, J. M. (2010).21st Century Auditing: Advancing Decision Support Systems to Achieve Continuous Auditing. Accounting Horizons.v.24, n. 2, p.297312. Koch, H. S. (1984). Auditing on-line systems: an evaluation of parallel versus continuous and intermittent simulation. Computers & Security, v.3, p.9-19. Li, S.-H.; Huang, S.-M.; Lin, Y.-C.G. (2007).Developing a continuous auditing assistance system based on information process models. Journal of Computer Information Systems. Rezaee, Z.; Sharbatoghlie, A.; Elam, R.; Mcmickle, L.P. (2002).Continuous auditing: building automated auditing capability. A Journal of Practice & Theory.v.21, n.1. Vasarhelyi, M. A.; Alles, M.(2008).The “now” economy and the traditional accounting reporting model: Opportunities and challenges. International Journal of Accounting Information Systems, v.9, n.4. Vasarhelyi, M. A.; Alles, M. G. and Kogan, A. (2002).Feasibility and economics of continuous assurance. A Journal of Practice & Theory, v.21, n.1. Vasarhelyi, M. A; Halper F. B. (1991).The continuous audit of online systems. A Journal of Practice & Theory, v.10, n.1, p.110-125. Vasarhelyi, M. A.; Alles, M. G.; and Kogan, A. (2004).Principles of analytic monitoring for continuous assurance. Journal of Emerging Technologies in Accounting, v.1, p.1-21. Watson, H. J. e Wixom, B. H. (2007).The current state of business intelligence. IEEE Computer Society, v.40, n.9.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 437-460

www.jistem.fea.usp.br

460 Silva, W.L. ,Júnior, J.R. de A.

Yeh, C.-H.e Shen, W.-C. (2010). Using continuous auditing life cycle management to ensure continuous assurance. African Journal of Business Management, v. 4, n.12, p.2554-2570. Yin, R. K. (2010). Estudo de caso– planejamento e métodos. 4 ed. Porto Alegre: Bookman.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.437-460

www.jistem.fea.usp.br

JISTEM - Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol. 11, No. 2, May/Aug., 2014 pp. 415-436 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200011

AUTOMATED TEXT CLUSTERING OF NEWSPAPER AND SCIENTIFIC TEXTS IN BRAZILIAN PORTUGUESE: ANALYSIS AND COMPARISON OF METHODS Alexandre Ribeiro Afonso Cláudio Gottschalg Duque University of Brasília (Universidade de Brasília–UnB), Brasília, DF, Brazil _____________________________________________________________________ ABSTRACT This article reports the findings of an empirical study about Automated Text Clustering applied to scientific articles and newspaper texts in Brazilian Portuguese, the objective was to find the most effective computational method able to cluster the input of texts in their original groups. The study covered four experiments, each experiment had four procedures: 1. Corpus Selections (a set of texts is selected for clustering), 2. Word Class Selections (Nouns, Verbs and Adjectives are chosen from each text by using specific algorithms), 3. Filtering Algorithms (a set of terms is selected from the results of the preview stage, a semantic weight is also inserted for each term and an index is generated for each text), 4. Clustering Algorithms (the clustering algorithms Simple KMeans, sIB and EM are applied to the indexes). After those procedures, clustering correctness and clustering time statistical results were collected. The sIB clustering algorithm is the best choice for both scientific and newspaper corpus, under the condition that the sIB clustering algorithm asks for the number of clusters as input before running (for the newspaper corpus, 68.9% correctness in 1 minute and for the scientific corpus, 77.8% correctness in 1 minute). The EM clustering algorithm additionally guesses the number of clusters without user intervention, but its best case is less than 53% correctness. Considering the experiments carried out, the results of human text classification and automated clustering are distant; it was also observed that the clustering correctness results vary according to the number of input texts and their topics. Keywords: Text Mining; Text Clustering; Natural Language Processing; Brazilian Portuguese; Effectiveness. _____________________________________________________________________________________________ Manuscript first received/Recebido em: 01/01/2013 Manuscript accepted/Aprovado em: 20/05/2014 Address for correspondence / Endereço para correspondência Alexandre Ribeiro Afonso is doctoral-degree student in the Information Science program (Faculdade de Ciência da Informação – FCI), University of Brasília (Universidade de Brasília–UnB), Brasília - DF, Brazil. E-mail: rafonso.alex@gmail.com Cláudio Gottschalg Duque works as professor and researcher in the Information Science program (Faculdade de Ciência da Informação – FCI), University of Brasília (Universidade de Brasília–UnB), Brasília - DF, Brazil. Campus Universitário Darcy Ribeiro, Faculdade de Ciência da Informação,Edifício da Biblioteca Central, Entrada Leste, Brasília, DF- Brazil. CEP: 70.919-970. Phone Number: 55(61)3107-2632. E-mail: klaussherzog@gmail.com Published by/ Publicado por: TECSI FEA USP – 2014 All rights reserved.

416 Afonso, A. R., Duque, C. G.

1. INTRODUCTION Automated text clustering systems have been developed and tested as an experimental and scientific activity. The purpose of text classification automation is to be as effective as humans when classifying texts in knowledge fields. A previous effective automatic clustering over a set of documents contributes to an effective automatic or manual information retrieval. The main difference between clustering and categorization systems is that the clustering systems do not utilize any formal knowledge (like ontologies or thesauri) for training previously the system; instead, it works as an unsupervised learning system (Manning at al., 2008). An automatic text clustering process could be divided in four main stages: Corpus Selection, Word Class Selections, Filtering Algorithms and Clustering Algorithms; during the experiments, we applied different procedures for each stage described to find the best combination of procedures which produced correct textual clustering by consuming less time, both for newspapers and scientific texts in Brazilian Portuguese. Clustering Algorithms have been developed for general use, for all languages, but they have been tested mainly for English, and many studies about text clustering, using corpora in English as input, have been described over the last decade. However, different natural languages could produce different levels of correctness in clustering results, since each natural language has specific structures and properties (such as morphological and syntax peculiarities) with different levels of complexity in their use (number of repetitions of words in newspaper texts, number of synonyms, use of idiomatic expressions, and terminologies). Some authors have described the impact of the linguistic characteristics over classification and information retrieval systems. For example, Rossel and Velupillai (2005) investigated the impact of using phrases in the vector space model for clustering documents in Swedish in different ways. Stefanowski and Weiss (2003) consider the problem of web search results clustering in the Polish language, supporting their analysis with results acquired from an experimental system named Carrot. Basic, Berecek and Cvitas (2005) argue that text processing algorithms and systems in English and other world languages are well developed, which is not the case with Croatian language, they affirm that the quality of input data strongly influences clustering and classification results. Another fact very important to be noted is that the format (news or scientific) and the content (History, Geography, Pharmacy, etc.) of the corpus could produce different results for clustering experiments. This verification was performed during our second experiment. All these cultural and linguistic particularities and the evident studies about the impact of the language over information retrieval systems make us to think about the scientific and newspaper communication in Brazil as having particular features. When analyzing the results of a clustering process, specifically for texts written in Brazilian

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

417

Portuguese, we can identify the best and not best procedures for each clustering stage, also observing their advantages and failures. Considering this observation, in future works, new procedures for text clustering can be proposed. The goal of our study was to verify whether an automated clustering process could create the correct clusters for two text corpuses: a scientific corpus having five knowledge fields (Pharmacy, Physical Education, Linguistics, Geography, and History) and a newspaper corpus having five knowledge fields (Human Sciences, Biological Sciences, Social Sciences, Religion and Thought, Exact Sciences). Therefore, we had two corpuses already classified by humans and we wanted to measure the effectiveness of the clustering process (clustering correctness and clustering time) by using the human classification as reference for correctness. Then, using a statistical method, we searched for the combinations of clustering stages that produced the best clustering correctness values and the shortest clustering time values. Through the following sections, we describe in detail each experiment stage (corpus selections, word class selections, filtering algorithms, and clustering algorithms), the procedures for each stage, the statistical methods adopted for clustering correctness and time measurement, statistical analysis, and results. 2. TEXT CLUSTERING IN BRAZILIAN PORTUGUESE: LITERATURE REVIEW 2.1 Text clustering approaches In this article, we focus on a non-hierarchical text clustering method as opposed to hierarchical text clustering approach. Non-hierarchical text clustering is applied when the goal is to produce text clusters which do not fit in a specific knowledge hierarchy; it means that each text is only inside a specific cluster, a text would not be grouped in two or more clusters at the same time (Markov & Larose, 2007). When a hierarchy is necessary to organize the texts, the hierarchical approach is able to group, for example, two related clusters inside a major cluster such as taxonomy. Some scientific fields like Medicine are structured in low level sub-fields and a hierarchical approach could be well applied. But, when the user needs to group the documents in major scientific areas (for example, Geography, Linguistics, Pharmacy, etc.), a nonhierarchical clustering method would be better applied because the fields do not have a high level of related terminological characteristics, or the fields/areas are independent. 2.2 Non-hierarchical text clustering in Brazilian Portuguese Our interest is to study the effect of the well-known text clustering technology (specifically, non-hierarchical text clustering methods) over a Brazilian digital repository where the texts are written in Brazilian Portuguese. Many possibilities for the four levels of a clustering experiment (corpus selections, word class selections, filtering algorithms, and clustering algorithms) were not tested yet for this language; so we execute new clustering experiments testing new corpora formats, filtering algorithms and using the well-known clustering algorithms.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

418 Afonso, A. R., Duque, C. G.

The three articles described below (Maia & Souza, 2010), (DaSilva et. al, 2004), and (Seno & Nunes, 2008) are closely related to our work. They test and evaluate the effectiveness of traditional non-hierarchic text clustering algorithms by using scientific and newspaper corpora in Brazilian Portuguese. The study reported by Maia and Souza (2010) is the most similar to our study. The study is about Automatic Text Categorization and Automatic Text Clustering in Brazilian Portuguese. The study regarding clustering methods aimed to compare the use of noun phrases and single terms as text representations for Simple K-Means clustering algorithm, trying to find the best linguistic representation. The researchers analyzed the effectiveness of the Simple K-Means clustering algorithm when clustering a corpus with 50 scientific texts about subareas of Information Science. The corpus was divided into 5 Information Science subareas (Historical and Epistemological Studies about Information; Knowledge Organization and Information Representation; Information Propagation, Mediation and Usability; Politics, Ethics and Information Economy; Management of Information Units) and the researchers aimed to evaluate the results from Simple K-Means when clustering texts indexed by noun phrases or terms. The manuscript also describes the same process for a newspaper corpus having 160 texts of 4 newspaper sections (informatics, tourism, world, vehicles). About the best clustering results, Maia and Souza (2010) describe 44% of clustering correctness using single terms from the scientific corpus, and 81% of clustering correctness using noun phrases from the newspaper corpus. They concluded that the use of noun phases is not better than the use of single terms as a representation for clustering tasks, since the correctness rate for noun phases and terms has an approximated percentage number, but the use of noun phases consumed a longer time. About the difference between text clustering and text categorization, considering the best results for correctness, they observed 8 percentage points difference between automatic text clustering and automatic text categorization for the scientific corpus, and 10 percentage points difference for the newspaper corpus, the text categorization process got the best values. A related study for Brazilian Portuguese was performed by DaSilva et al. (2004). The researchers propose and evaluate the use of linguistic information in the preprocessing phase of text mining tasks (categorization and clustering). They present several experiments comparing their proposal for selection of terms based on linguistic knowledge with usual techniques applied in the field. The results show that part of speech information (in this paper, we use the term word classes) is useful for the preprocessing phase of text categorization and clustering, as an alternative for stop words and stemming. We could also cite the study of Seno and Nunes (2008) as a related work. The paper presents some experiments on detecting and clustering similar sentences of texts in Brazilian Portuguese. They propose an evaluation framework based on an incremental and unsupervised clustering method which is combined with statistical similarity metrics to measure the semantic distance between sentences. Experiments show that this method is robust even to treat small data sets. It has achieved 86% and 93% of F-measure and Purity, respectively, and 0.037 of Entropy for the best case.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

419

3. AUTOMATED TEXT CLUSTERING: DESCRIPTION OF THE EXPERIMENTS’ STAGES Our study was performed by analyzing the statistical results from four text clustering experiments. The four experiments had the same architecture; it means the four experiments are composed by the same sequential stages. In this section, we describe the possibilities for the four sequential stages: Corpus Selections, Word Class Selections, Filtering Algorithms, and Clustering Algorithms. Corpus Selections: We chose two textual databases to extract the newspaper and the scientific corpus. The scientific corpus was taken from five scientific journals published by the digital library of Federal University of Goiás (UFG) in Brazil. We got the articles from five scientific fields (Pharmacy, Physical Education, Linguistics, Geography, and History). The text choices from the scientific digital library were a random process. The newspaper corpus was taken from the "Lácio-Web Textual Database" described by Aluísio and Almeida (2006). It is produced by NILC (Núcleo Interinstitucional de Linguística Computacional), a Computational Linguistics research group in Brazil. The texts from Lácio-Web database are classified in five areas (Human Sciences, Biological Sciences, Social Sciences, Religion and Thought, Exact Sciences). Lácio-Web stores texts from newspapers that are currently active in Brazil. The text choices from the textual database were a random process. The content of the newspaper texts was not modified after corpus selection, but a text partition was executed for the scientific corpus after corpus selection. Since the scientific texts are larger texts, we took from each scientific text: title, abstract, keywords, and the first page/column of the introduction; the first page was selected since it was observed that in scientific texts the introduction, generally, fills no more than one or two pages/columns (considering the scientific fields analyzed), this choice permits to keep a small set of terms but considering the content of the introduction. An initial conjecture was that the terms which identify the topic are more frequent in these first parts of the scientific text, this verification is part of the experiments done. Although some texts from the Lácio-Web Database can be classified in more than one area, we did not permit text replicas in our corpus. When a text was found inside two or more clusters in our corpus, it was replaced by another one, but the text chosen could still be replicated by many clusters in Lácio-Web database. The original topics of the scientific articles are chosen according to the scope descriptions of the journals in their web pages (there is a section "Foco e Escopo" for each journal which describes the topics of interest). The scientific database is found at (http://www.revistas.ufg.br/) and the newspaper Lácio-Web database can be found at (http://www.nilc.icmc.usp.br/lacioweb/). Both formats (newspaper and scientific texts) have public access and they are commonly found in libraries, and since our goal was to evaluate the current technology over digital libraries, we decided to test these specific text formats. We worked with at

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

420 Afonso, A. R., Duque, C. G.

most five knowledge areas for each format during the experiments; if we chose more than five knowledge areas it would create a very large study for a single research. Another reason for this number is that we divided the original set of texts in two corpuses (having three and five knowledge areas) to verify whether, when the number of knowledge areas and the number of texts increase, the clustering effectiveness decreases. The criterion for choosing the knowledge areas to construct the corpora is the approximation of the knowledge areas and their distances; it means we decided to choose two areas inserted in the same major area (e.g. History and Geography) and other two areas not directly related (e.g. Physical Education and Linguistics) and one more area was chosen randomly. The same criterion was set for the newspaper texts (Social Sciences and Religion and Thought) and (Human Sciences and Biological Sciences). The two sets of data produced after this manual processes were the experiment corpora that follow the original human classifications from the two databases. Word Class Selections: We first applied (for each text from both corpuses) a POSTagger algorithm (Part-of-Speech tagger); it inserts tags (word class tags like Noun, Verb, Adverb, Adjective, etc.) to every word inside a text. The POS-Tagger used is trained by the researchers of the NILC institute, as described by Aires (2000), the MXPOST Tagger application (Ratnaparkhi, 1996) was used to produce this Brazilian Portuguese Tagger. This POS-Tagger was chosen because the taggers for (Nouns, Adjectives and Verbs) are inserted, and the tagger does not differentiate the hierarchies of the nouns, it also was trained by using a Brazilian Portuguese corpus. Moreover, the tagger was developed and tested by natural language processing specialists from NILC. After applying the POS-Tagger algorithm for each text of the corpus and selecting Nouns, Verbs and Adjectives, we applied a stemming algorithm for each tagged text. The stemming process was performed by using an application developed by NILC; it is described by Caldas (2001). The stemmer follows Porter's algorithm and works for Brazilian Portuguese by identifying the stem of words by incrementally removing their suffix/termination. After tagging and stemming, each text from the scientific and newspaper corpus is represented only by their stems and their tags (only Nouns, Verbs and Adjectives). This stemmer was chosen because it was developed specifically for Brazilian Portuguese, and it was developed by natural language processing specialists from NILC. Filtering Algorithms: In this experiment stage, we applied some algorithms for creating a text index for each text. The index must contain the main stems from the previous word class selections stage and it has to represent the text by holding its semantic meaning. So, the index produced by this stage is a reduced set of stems also having a semantic weight for each stem. The semantic weight algorithm executes a mathematical function that returns the stem's semantic weight according to its semantic importance inside the text and the entire corpus. For the four experiments, we applied the mathematical function IDF (Inverse Document Frequency)-Transform (Markov & Larose, 2007):

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

421

IDF-T = fij*log (number of documents/number of documents with stem i) (I) fij is the frequency of the stem i in document j. The Weka toolkit is a free software for data mining and text mining tasks, and we used Weka software to apply the IDF-T. After inserting a semantic weight (IDF-T) for each stem of each text, we can apply one of three procedures for stem selections. The first procedure is to use the same set of stems returned by the word class selections stage without intervention. It means the index produced by the filtering algorithms stage is simply the set of stems selected during the word class selections: (nouns), (nouns, verbs), (nouns, adjectives), (verb, adjectives), (nouns, adjectives, verbs), or (nouns, adjectives, verbs - without tags) with their IDF-T semantic weights. The second filter procedure possible is to use an intelligent algorithm based on the Genetic Search Metaheuristic (Goldberg, 1989) combined with Correlation-based Feature Subset Selection (it is named cfsSubSetEval by Weka) for selecting the stems, both implemented by Weka. The third option is to filter the set of stems coming from the word class selections stage by using a frequency filter algorithm; this frequency filter algorithm selects the stems according to the stems’ frequencies adopted by the researcher, for example if the researcher chose (frequency: 2) only the stems having at least two occurrences inside the corpus could be selected to the output index. These filtering procedures for selecting terms were chosen since they work over different paradigms (no specific filtering, an intelligent filtering, or a frequency filtering), the objective was to verify the impact of each one during the clustering experiments. The indexes produced during this stage represent the input texts as a set of selected terms (text indexes) and they are clustered by the clustering algorithms during the next stage. Clustering Algorithms: The clustering algorithms are executed in this final stage. Since the clustering algorithms do not receive any extra knowledge (like a hint for the topics or vocabulary) about the texts to be classified, it only works on the texts’ content, and the clustering algorithms must preview the topic for each cluster, or even the number of clusters. A few clustering algorithms such as SKM (Simple K-Means) described by Manning and Schütze (1999), and sIB (Sequential Information Bottleneck) described by Slonin et al. (2002), are algorithms that ask for the number of clusters from the user. Other algorithms such as EM (Expectation Maximization) (Manning & Schütze, 1999), Evolutionary Searches (Jones at al., 1995), or Neural Networks Architectures (Kohonen, 1997), (Kohonen, 1998) can try to guess the number of clusters and the topic for each cluster, but guessing the number of clusters is a more difficult algorithmic task. Each clustering algorithm may find a different result, since they could apply different mathematical strategies and heuristics to compare the texts (represented by indexes) when clustering. Algorithms can use formal knowledge of the language (as grammar structures or textual features) when executing a clustering process, therefore natural languages having a high level of scientific description would get better results for text clustering. The Data Mining toolkit Weka already has the three clustering algorithms we chose for the experiments (EM, SKM, and sIB), so we decided to use this tool again for the clustering algorithms stage of the experiments. Many options of running can be

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

422 Afonso, A. R., Duque, C. G.

chosen before starting the algorithm execution; we can choose an iteration number, the number of groups for clustering, and other restrictive alternatives for each algorithm. The three algorithms we tested make linear and excluding clusters, which means they are non-hierarchical. After describing each stage, we can set a final architecture for the experiments, the figure below shows the final model adopted for the four experiments performed. The evaluation methods for measurement are described in the next section.

Figure 1: Experiment Stages and measurement

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

423

4. EVALUATION METHODS 4.1 Clustering correctness metrics The usual metrics for clustering correctness are F-measure, Percentage Values of Correctness, Purity and Entropy. The usual metrics evaluate clustering algorithms' performance and the quality of the resulting clusters. F-Measure has been applied to measure the quality of the entire clustering process, while Purity and Entropy are used to measure the quality of the resulting clusters, and Percentage Values of Correctness is applied to measure the errors of the clustering results considering a human classification as reference, Percentage Values of Correctness returns the exact number of clustering errors from each cluster. Song and Park (2006) only applied F-Measure, Seno and Nunes (2008) uses three metrics (F-measure, Purity and Entropy), and Maia and Souza (2010) uses Percentage Values of Correctness as metric, we notice the choice of each metric depends on the objectives of the experiment. Our decision was to use Percentage Values of Correctness as metric, since we wanted to compare the clustering results returned by the clustering algorithms and the previous human classification, the exact number of clustering mistakes from each experiment was necessary to choose the best clustering experiment. We also used an additional metric for measuring the number of deviated clusters, a Deviation Number (DN) which identifies the exact number of clusters created more than the expected number of clusters or less than the expected number of clusters. This metric value (DN) is calculated only for the EM algorithm since only this clustering algorithm guesses the number of clusters when executing a clustering process. We realized that these two values (Percentage Values of Correctness and a Deviation Number - DN) are more informative than using a single statistical value about the distribution of the documents over the clusters created, as returned by (Fmeasure, Purity and Entropy metrics); the results of the two metrics described is more informative (for this specific set of experiments) then the usual averages since they return two different exact numbers and not a single average from a clustering result. The result analysis is performed according to Weka outputs. For each clustering algorithm execution, Weka provides the output of n clusters (0 to n-1) and m texts classified (0 to m-1) for each cluster. We can see the clustering results through the result window as Cartesian Coordinates (x, y) and colors for each cluster. The next figure 2 below shows the result for the sIB algorithm execution for 3 clusters:

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

424 Afonso, A. R., Duque, C. G.

Figure 2: Weka image for sIB clustering algorithm output The x-axis holds the number of the document; this sample experiment has 36 text documents. By our configuration, we know that the 0-11 numbers represent the documents about cluster 1 (for example, Pharmacy); numbers 12-23 represent cluster 2 (for example, Physical Education) and cluster 3 (for example, Linguistics) is represented by numbers 24-35. The y-axis holds the corpus items (in this case, stems), and the symbol "*" indicates both a document (0-35) and a Boolean value (true or false) for a stem on the y-axis. If the document has the stem shown on the y-axis then the symbol "*" is aligned according to the y-axis. The color of the symbol "*" indicates the cluster where the document is found. Each cluster produced by the sIB, EM or SKM algorithm execution will get a specific color. The percentage values of correctness are calculated by analyzing the results displayed by figure 2, it shows us a Weka experiment result using the sIB algorithm, for 3 clusters (Pharmacy, Physical Education, and Linguistics), with 12 documents queued for each area (0 to 35), as shown by the x-axis. Each color represents a cluster generated by the system (sIB asks the user for the number of clusters to be generated). Cluster 0 is blue, cluster 1 is red, and cluster 2 is green. We consider the last 12 elements as green, with 7 mistakes; the 12 elements in the middle are blue, having 2 mistakes; and the 12 elements at the beginning are red, with 6 mistakes. The error summation is 15, so the experiment correctness percentage is 58.3% of correct classifications. We apply this calculus to all the experiments performed.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

425

For a clustering experiment using the EM algorithm we use two metrics: we check whether the number of clusters generated by the EM algorithm execution is correct or, if not correct, we calculate a deviation number (DN) given by the absolute value function (it always returns a number equal or greater than zero): DN = ( | Number of Expected Clusters - Number of Output Clusters | ) (II) Then we calculate the percentage values of correctness as described for the sIB and SKM algorithms. 4.2 Clustering time metrics The time unit chosen for clustering time measurement is minutes, since the executions are long, and they could take from seconds to hours to complete. Minutes are an intermediate metric, and for this reason, we decided to use this time unit. When the time taken by an execution is broken, like 2 minutes and 35 seconds, we chose to extend the number to the next unit of minutes, but if the past seconds are less than 30, we round down to the nearest minute unit. For example, 2 min 29 sec. becomes 2 minutes, but 2 min. 30 sec. becomes 3 min. It is difficult to measure and compare exact time, since the computational conditions could vary even when the machine architecture, hardware configuration, operating system, and compiler are the same. For example, let's suppose we expect the time results for two machines having the same machine architecture and running the same clustering algorithm for the same experiment, but for any reason the number of processes running on machine one is greater than machine two. As a consequence the time results will have different values when clustering is finished, since machine one needs to control more processes than machine two. For this reason, the time values we got here are just an approximation under a few specific conditions. For these experiments, we use a machine having a Pentium Dual Core T4500 Intel processor. It has 2 GB Main Memory; the Operating System is Microsoft Windows XP Professional 2002 SP 3; the Weka version is 3.6.3; and the Java Development Kit (JDK) installed is jdk1.6.0_04. We took notes only for the time consumed by Weka to build the clusters. It means we just measured the time taken by the clustering algorithms stage: SKM, sIB, and EM clustering algorithms. 5. RESEARCH QUESTIONS Question 1: Considering the four experiments performed, and their different configurations for the four stages, what is the experiment which produces best results (the one that produces the best clustering correctness value and consumes less time) having a scientific corpus as input? Question 2: Considering the four experiments performed, and their different configurations for the four stages, what is the experiment which produces best results

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

426 Afonso, A. R., Duque, C. G.

(the one that produces the best clustering correctness value and consumes less time) having a newspaper corpus as input? 6. EXPERIMENTSâ&#x20AC;&#x2122; DESCRIPTION AND RESULTS 6.1 Experiment 1 6.1.1 Configurations of the experiment 1A (newspaper corpus) Corpus Selection: Newspaper texts from LĂĄcio-Web corpus, related to Biological Sciences, Exact Sciences, and Human Sciences. 15 texts from each area. Total: 45 texts. Word Class Selections: We experimented different sets of word tags (4 options kept the word class tags and 1 option did not use the tags): (nouns), (nouns, verbs), (nouns, adjectives), (nouns, adjectives, verbs), (nouns, adjectives, verbs - without the tags). Filtering Algorithms: 2 options: A Weka intelligent filter named Genetic Search with the method for evaluating stems named cfsSubSetEval. We kept the default values for (GeneticSearch/ cfsSubSetEval). Another try with no filter. The formula for Lexical Item Weights was only IDF-T. Clustering Algorithms : EM, SKM and sIB algorithms. Notice there are 30 executions for each possible combination above (5 Sets of Tags * 2 Filters Procedures * 3 Clustering Algorithms). 6.1.2 Configurations of the experiment 1B (scientific corpus) Corpus Selection: Scientific Journals from UFG Digital Library from 3 different journals, about Pharmacy, Physical Education, and Linguistics. 12 papers from each journal. Total: 36 texts. Word Class Selections: We tried many different sets of tags (4 options kept the word class tags and 1 option without the tags): (nouns), (nouns, verbs), (nouns, adjectives), (nouns, adjectives, verbs), (nouns, adjectives, verbs - without the tags). Filtering Algorithms: 2 options: A Weka intelligent filter named Genetic Search with the method for evaluating stems named cfsSubSetEval. We kept the default values for (GeneticSearch/ cfsSubSetEval). Another try with no filter. The formula for Lexical Item Weights was only IDF-T. Clustering Algorithms : EM, SKM and sIB algorithms. Notice there are 30 executions for each possible combination above (5 Sets of Tags * 2 Filters Procedures * 3 Clustering Algorithms). For both Experiments 1A and 1B, the result having the best value of correctness for each clustering algorithm is chosen as the best (only one combination for each algorithm is chosen).

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

427

6.2 Experiment 2 Experiment 2 aimed to verify the clustering correctness percentage when we inserted more input texts and clusters; a more diverse corpus was compiled. We verified how the percentage correctness error rate increases when adding more textual elements. For this experiment, we only tested the combinations that reached the best values for Experiments 1A and 1B, since we wanted to verify whether the best values of clustering correctness would be kept from the experiments 1A and 1B, and how much additional time it would consume. It means we redid the experiments 1A and 1B, but having 5 knowledge areas and 5 clusters and choosing new documents for the five knowledge areas. 6.2.1 Configurations of the experiment 2A (newspaper corpus) For Experiment 2A, which works with newspaper texts, we added two additional sets: Social Sciences, and Religion and Thought texts (15 additional texts for each area). The set of POS-Tags and filters were the same from the best values obtained during Experiment 1A. Therefore, the algorithms had to produce 5 clusters instead of 3 clusters, and the total number of texts was 75 texts for the newspaper corpus. 6.2.2 Configurations of the experiment 2B (scientific corpus) For Experiment 2B, which works with scientific texts, we added History and Geography texts (12 additional texts for each area). The set of POS-Tag and filters were the same from the best values gotten during Experiment 1B. Therefore, the algorithms had to produce 5 clusters instead of 3 clusters, and the total number of texts was 60 texts for the scientific corpus. The total number of executions for this experiment is 6: (1 filter * 3 algorithms * 1 tag set) = 3 executions for each corpus. 6.3 Experiment 3 This experiment verified the effect of taking the Nouns out of the text indexes. Our goal was to check the importance of the Nouns for text clustering. If the importance of the Nouns for clustering is high, their absence will produce poorer clusters. We again modified a few features from Experiment 1. For this experiment, two corpuses were used as before, a newspaper corpus, the same from Experiment 1A, and a scientific corpus, the same from Experiment 1B. From Experiment 1A and 1B, all the configurations described were kept, except the POS-Tags. Here we only used the tag set: (Verb, Adj); so we kept the configurations for corpus selections, filter algorithms, clustering algorithms but this time we tested only Verbs and Adjectives as tags for the stage word class selections. The total number of executions for this experiment is 12 (2 filters * 3 algorithms * 1 tag set) = 6 executions for each corpus. The result having the best value of correctness for each clustering algorithm was chosen as the best.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

428 Afonso, A. R., Duque, C. G.

6.4 Experiment 4 This experiment used another kind of filtering procedure for the filter algorithms stage, so we tried a filter based on stem frequencies. It means that instead of using the text indexes produced during Experiment 1, this time we selected the main stems from the texts to be clustered according to their frequency inside the corpus. The goal was to verify whether the filter’s variation causes a notable effect over clustering performance (correctness and time). We tried different frequencies 2, 3, 4, 5 for each term of the corpus for this new filter, since we were trying to find the best one. During these executions we only chose two sets of tags: (Nouns) and (Nouns, Verbs, Adjectives – without tags) for the Word Class Selections stage. These word classes were chosen since, initially, we had the hypothesis that (Nouns) were the best alternative to identify a topic from a text, and the set (Nouns, Verbs, Adjectives - without tags) has been applied to many works for text mining, indexing, and clustering. We performed these new tests for the newspaper corpus (Experiment 4A) and also for the scientific corpus (Experiment 4B). When we specify we tried different frequencies 2, 3, 4, 5 for the experiment, it means that we only considered stems having frequency 2, 3, 4, or 5 inside the entire corpus for each execution test. The number of execution tests for each corpus was ((4 possible frequencies for filtering * 3 clustering algorithms * 2 tag sets) = 24 possibilities), or 48 executions considering both newspaper and scientific corpus. The result having the best value of correctness for each clustering algorithm was chosen as the best. 7. EXPERIMENTAL ANALYSIS AND RESEARCH ANSWERS The three pairs of charts below summarize the best results of clustering correctness and time achievement for experiments (1-4)A (newspaper corpus) and (14)B (scientific corpus), which we described in the last section. In the first graphic pair, we use a percentage notation “%” for clustering correctness, in the second pair a time notation in minutes. The last pair of charts shows the deviation numbers (DN) for the EM algorithm. The blue charts are about Experiments A (newspaper corpus) and red graphics about Experiments B (scientific corpus). Notice the results are always only for the last experiment stage (Clustering Algorithms), so the results for time and correctness are always from the sIB, SKM and EM algorithms.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

429

Figure 3: Two charts show the clustering correctness values gotten by each algorithm for each experiment (1-4) described. Percentage values of correctness on y-axis and the experiment name on x-axis.

Figure 4: Two charts show the time consumed by each algorithm. Time in minutes on y-axis and the experiment name on x-axis.

Figure 5: Two charts show the Deviation Number (DN) only for the EM algorithm. Units of deviation on y-axis and the experiment name on x-axis.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

430 Afonso, A. R., Duque, C. G.

Question 1 - Notable patterns about the scientific corpus (experiments: 1B, 2B, 3B and 4B – red graphics) Best stages’ combination for the scientific corpus considering correctness and time: 77.8% correctness in 1 minute by the fourth experiment. Configuration for the best result (considering the best rate of correctness and less time): Corpus selections: A corpus having three not related topics, Pharmacy, Linguistics and Physical Education. Word class selections: (Nouns + Verbs + Adjectives - No Tags). Filtering algorithms: (Frequency Filter: Frequency 5). Clustering algorithm: sIB. Question 2 - Notable patterns about the newspaper corpus (experiments: 1A, 2A, 3A and 4A – blue chart) Best stages’ combination for the newspaper corpus considering correctness and time: 68.9% correctness in 1 minute by the fourth experiment. Configuration for the best result (considering the best rate of correctness and less time): Corpus selections: A corpus having three not related topics, Biological Sciences, Exact Sciences, and Human Sciences. Word class selections: (Nouns) Filtering algorithms: (Frequency Filter: Frequency 3) Clustering algorithm: sIB For the newspaper experiments, we must observe that a few texts selected for tests can be found inside two or more groups from the Lácio-Web Database. The corpus for Experiments 1, 3, and 4 has 31.1% of the texts previously classified inside more than one group. The corpus for Experiment 2 has 9.3% of the texts previously classified inside more than one group. 8. CONCLUSION This study tried to determine certain performance patterns when executing automated text clustering. We verified whether the best results found for automated text clustering and human text classification are approximated, we chose a realistic, experimental, and statistical point of view for finding clustering correctness and time values. Many works on this topic can be found for the English language, but we could

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

431

not find many works for Brazilian Portuguese, so this was the motivation for our research. The first problem about evaluation of text clusters produced by a clustering algorithm is the concept of correctness: what is correct clustering? Or, what is a good cluster? Is the best clustering process the one that got the best time, or the best grouping or both? We accepted the predefined classified databases used as having correct classification, and they were produced by humans, so this classification follows a specific criterion, but is this specific criterion the best one? What is the best choice to the user: a system able to find the correct number of clusters having a low rate of clustering correctness, or having a high rate of clustering correctness over additional clusters? We can see the problem of evaluating text clustering is something relative, and the concept of correctness and quality for clustering depends on one's expectations. During this study, we chose a specific criterion for evaluation as we described during the experimentsâ&#x20AC;&#x2122; description. It was noticed that there is a combinatorial property for this kind of investigation, and the combinatorial choices are broad, with many different trees of possible investigation for the experiment stages: corpus selections, word class selections, filtering algorithms, and clustering algorithms. So we decided to investigate a few trees and specific branches for these trees, using the current linguistic tools available for Brazilian Portuguese. We see the possible investigations are very broad, and simple details during the stages of the experiments can produce very different results. Although we did 126 clustering tests during the four experiments, we investigated only a small part of the possibilities, under some exact conditions. So many attempts can still be performed. Considering the results, regularity can be observed between the newspaper and the scientific experiments: the sIB algorithm always got the best clustering correctness values for both corpuses. We can notice the rate of clustering correctness for newspaper experiments is equal to or lower than the values obtained for the scientific corpus for all tests, except for Experiments sIB-2A, EM-3A, and EM-4A. One possible reason for this best result is the scientific vocabulary, since it is formal and rich, and it uses specific language which gives better clues for clustering algorithms. Although we have gotten a reasonable result for the tuple (scientific texts, frequency filter, Nouns + Verbs + Adjectives - without tags, sIB algorithm) as described, the sIB algorithm asks for the number of clusters to be created and, in most cases, the user does not know the nature of the texts and topics or even the number of clusters; moreover sIB algorithm got a reasonable result only when producing three clusters for the scientific corpus. Some algorithms (like the EM algorithm) try to find the number of cluster and their terminological features, but EM does not yet have a high level of correctness, and much work must be done in this way to improve the results. The second experiment (which works with five clusters) got a very worse result. We can see from Experiments 1 and 2 (figure 3) that the differential values of clustering correctness between the columns (Experiment 1A and Experiment 2A), for each clustering algorithm, have lower values than the differential values between the

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

432 Afonso, A. R., Duque, C. G.

columns (Experiment 1B and Experiment 2B) for each clustering algorithm. Perhaps it happens because of the scientific corpus chosen, since we added History and Geography texts for the new Experiment, 2B. So, we suspect that these two scientific fields have a similar vocabulary, which would cause an algorithmic clustering confusion over these two clusters, or that the two fields do not have a very formal or technical vocabulary powerful enough for clear text discrimination in relation to the other fields. Therefore, this fact opens a new hypothesis for the higher percentage difference between the columns (Experiment 1B and Experiment 2B). Something important to observe is the fact that texts have a multi-classification nature and according to Ranganathan (1967), a textual topic could be analyzed from many different viewpoints (classes and facets). Even though works and studies about automatic text classification and clustering have been using a deterministic classification (each document inside only one group), a more realistic classification should permit a document to be inserted into more than one group, at least for some knowledge fields from the newspaper corpus. We realized that scientific texts are better classified using a deterministic approach than newspaper texts, since most of time a formal vocabulary is present for each scientific field. This formal language could not always be identified for newspaper texts, so an experimental possibility is to redo the tests considering a multi-clustering possibility for each text, but we should also consider that a multi-clustering approach could generate difficult searching and maintenance for documents, at least for databases having many clusters and having the same document inside many clusters. Without getting a semantic or pragmatic level of analysis it would be hard to produce a good and practical clustering near to human classification. Therefore, we believe that general linguistic tools are important for clustering success. A Wordnet database (Fellbaum, 1998) for Brazilian Portuguese and other general language descriptions would play a very important role in acceptable clustering, at least for informative newspaper texts. The evolution of the algorithms, scientific natural language descriptions and Bibliometrics is a key for text clustering improvement.

REFERENCES Aires, R. V. X. (2000). Implementação, Adaptação, Combinação e Avaliação de Etiquetadores para o Português do Brasil. [Implementation, Adaptation, Combination and Evaluation of Brasilian Portuguese Taggers.] Unpublished MsC Thesis. Universidade de São Paulo, São Paulo, Brazil. Aluísio S. M., & Almeida G. M. B. (2006). O que é e como se constrói um corpus? Lições aprendidas na compilação de vários corpora para pesquisa lingüística. [What is a corpus and how to build a corpus? Lessons learned during the compilation of various corpora for linguistic research.] Calidoscópio, 4(3), 155-177.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

433

Basic, B. D., Berecek B., & Cvitas A. (2005). Mining textual data in Croatian, in Proceedings of the 28th International Conference, MIPRO 2005, Business Intelligence Systems. (pp. 61–66). Bezerra, G. B., Barra, T. V., Ferreira, H. M., & Von Zuben, F. J. (2006). A hierarchical immune-inspired approach for text clustering. Selected papers based on the presentations at the 2006 conference on Information Processing and Management of Uncertainty, IMPU, Paris, France. (pp. 131-142). Biderman, M. T. C. (2001). O Português Brasileiro e o Português Europeu : Identidade e contrastes. [The Brazilian Portuguese and European Portuguese: Identity and contrasts.] Revue belge de philologie et d'histoire, 79 (3), 963-975. Caldas Junior, J., Imamura, C.Y.M., & Rezende, S.O. (2001). Avaliação de um Algoritmo de Stemming para a Língua Portuguesa. [Evaluation of a Stemming Algorithm for Portuguese Language.] The Proceedings of the 2nd Congress of Logic Applied to Technology, São Paulo, Brazil. (pp. 267-274). São Paulo, Brazil: SENAC/Plêiade. Camargo, Y. B. L. (2007). Abordagem lingüística na classificação de textos em português. [A linguistic approach to the classification of texts in Portuguese.] Unpublished MsC Thesis. COPPE - Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil. DaSilva, C. F., Vieira, R., Osório, F. S., & Quaresma, P. (2004). Mining Linguistically Interpreted Texts. In Proceedings of the 5th International Workshop on Linguistically Interpreted Corpora. Fellbaum, C. (1998). WordNet. An electronic lexical database. Cambridge, MA: MIT Press. Furlanetto, M. M. (2008). Neological Formations in Brazilian Portuguese: a Discursive View. Fórum Lingüístico, 5 (2), 1-22, Florianópolis, Brazil. Goldberg D. E. (1989). Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley. Hall M. A. (1998). Correlation-based Feature Subset Selection for Machine Learning. Hamilton, New Zealand. (University of Waikato Ph.D. dissertation). IBGE (2000). Brasil: 500 anos de povoamento [Brazil: 500 years of settlement]. Rio de janeiro: IBGE. Jones, G., Robertson, A. M., Santimetvirul, C., & Willett, P. (1995). Non-hierarchic document clustering using a genetic algorithm. Information Research, 1(1), Retrieved 15 April, 2012 from http://InformationR.net/ir/1-1/paper1.html. Hammouda, K. M., & Kamel. M. S. (2004). Efficient phrase-based document indexing for web document clustering. IEEE Transactions on Knowledge and Data Engineering, 16(10), 1279–1296. Kohonen, T. (1997). Self-Organizing Maps. 2nd ed., Berlin: Springer-Verlag.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

434 Afonso, A. R., Duque, C. G.

Kohonen, T. (1998). Self-Organization of Very Large Document Collections: State of the Art. Proceedings of ICANN98, the 8th International Conference on Artificial Neural Networks, Skövde, Sweden, 2-4 September, 1998, 8(1), 65-74: Springer. Maia, L. C., & Souza, R. R. (2010). Uso de sintagmas nominais na classificação automática de documentos eletrônicos. [The use of noun phrases in automatic classification of electronic documents.] Perspect. Ciênc. Inf., 15(1), 154-172. Manning, C. D, Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi: Cambridge University Press. Manning, C. D., & Schütze, H. (1999). Foundations of Statistical Natural Language Processing. Cambridge, Massachusetts: The MIT Press. Markov, Z., & Larose D. T. (2007). Data Mining the Web: Uncovering Patterns in Web Content, Structure and Usage. Hoboken, New Jersey: John Wiley ; Sons, Inc. Palmeira, E., & Freitas, F. (2007). Ontologias detalhadas e classificação de texto: uma união promissora. [Detailed ontologies and text classification: a promising union.] ENIA 2007: VI Encontro Nacional de Inteligência Artificial. Rio de Janeiro, Brazil, July, 03-06, 2007. Rio de Janeiro: Instituto Militar de Engenharia. Ranganathan, S. R. (1967). Prolegomena to Library Classification. London: Asia Publishing House. Ratnaparkhi, A. (1996). A Maximum Entropy Model for Part-of-Speech Tagging. Proceedings of the First Empirical Methods in NLP Conference. University of Pennsylvania, May 17-18, 1996. (pp. 133-142). Reis, João José (2000). Presença Negra: conflitos e encontros. In Brasil: 500 anos de povoamento [Black Presence: conflicts and encounters]. Rio de Janeiro: IBGE, 2000. pp: 91. Rossel, M., & Velupillai, S. (2005). The Impact of Phrases in Document Clustering for Swedish. Proceedings of the 15th NODALIDA conference, NoDaLiDa 2005, Joensuu, Finland. (pp.173-179). Seno, E. R. M., & Nunes, M. D. V. (2008). Some Experiments on Clustering Similar Sentences of Texts in Portuguese. In Teixeira, A., StrubeDeLima, V. L., CaldasDeOliveira, L., Quaresma, P. (Eds.), Lecture Notes in Artificial Intelligence, Vol. 5190. 8th International Conference on Computational Processing of the Portuguese Language, PROPOR 2008, Aveiro, Portugal, September 08-10, 2008. (pp. 133-142). Berlin, Germany: Springer-Verlag. Silva. A. S. (2006). Sociolinguística cognitiva e o estudo da convergência/divergência entre o Português Europeu e o Português Brasileiro. [Cognitive Sociolinguistics and the study of convergence / divergence between European Portuguese and Brazilian Portuguese.] Veredas :Revista de Estudos Lingüísticos, 10 (2006): Universidade Federal de Juiz de Fora. Slonin, N., Friedman N., & Tishby, N. (2002). Unsupervised document classification using sequential information maximization. Proceedings of the 25th International ACM

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

Automated Text Clustering of Newspaper And Scientific Texts in Brazilian Portuguese: Analysis and Comparison of Methods

435

SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, August 11-15, 2002. (pp. 129-136). New York: ACM Press. Song W., & Park S. C. (2006). Genetic Algorithm-based Text Clustering Technique. In Licheng Jiao, Lipo Wang, Xinbo Gao, Jing Liu, Feng Wu (Eds.), Lecture Notes in Computer Science, Vol. 4221. Advances in Natural Computation, Second International Conference, ICNC 2006, Xi'an, China, September 24-28, 2006. (pp. 779-782). Berlin: Springer-Verlag. Stefanowski, J., & Weiss, D. (2003). Web search results clustering in Polish: experimental evaluation of Carrot. Advances in Soft Computing, Intelligent Information Processing and Web Mining, Proceedings of the International IIS: IIPWM´03 Conference, Zakopane, Poland, vol. 579 (14). (pp. 209-22). Viera, A.F.G., & Virgil, J. (2007). Uma revisão dos algoritmos de radicalização em língua portuguesa. [A review of stemming algorithms for Portuguese Language.] Information Research, 12(3), paper 315. Retrieved 15 April, 2012 from http://InformationR.net/ir/12-3/paper315.html. Witten I. H., & Frank E. (2005). Data Mining: Practical Machine Learning Tools and Techniques, 2º Ed. Amsterdam, Boston, Heidelberg, London, New York, Oxford, Paris, San Diego, San Francisco, Singapure, Sydney, Tokyo: Elsevier.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

436 Afonso, A. R., Duque, C. G.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp.415-436

www.jistem.fea.usp.br

JISTEM Revista de Gestão da Tecnologia e Sistemas de Informação Journal of Information Systems and Technology Management Vol. 11, No. 2, 2014, p.461-510 ISSN online: 1807-1775 DOI: 10.4301/S1807-17752014000200013

RESULTADOS DO 11º CONTECSI USP – CONGRESSO INTERNACIONAL DE GESTÃO DA TECNOLOGIA E SISTEMAS DE INFORMAÇÃO OUTCOMES OF THE 11th CONTECSI USP – INTERNATIONAL CONFERENCE ON INFORMATION SYSTEMS AND TECHNOLOGY MANAGEMENT Edson Luiz Riccio Universidade de São Paulo, São Paulo, Brasil Marici Cristine Gramacho Sakata TECSI FEA USP, Universidade de São Paulo, São Paulo, Brasil Nelma Terezinha Zubek Valente Universidade Estadual de Ponta Grossa, Paraná, Brasil Ligia Capobianco Universidade de São Paulo, São Paulo, Brasil

ABSTRACT In this document, we wish to describe statistics, data and the importance of the 11th CONTECSI – International Conference on Information Systems and Technology Management, which took place in the University of São Paulo, during the dates of May 28th to 30th and was organized by TECSI/EAC/FEA/USP. This report presents statistics of the 11th CONTECSI, Goals and Objectives, Program, Plenary Sessions, Doctoral Consortium, Parallel Sessions, Honorable Mentions and Committees. We would like to point out the huge importance of the financial aid given by CAPES, CNPq, FAPESP, as well as the support of FEA USP, ANPAD, AIS, ISACA, UNINOVE, Universidade do Porto, Rutgers School/USA, São Paulo Convention Bureau and CCINT-FEA-USP. Keywords: CONTECSI; Report; International Information Systems; Observatory.

Conference; Technology Management;

____________________________________________________________________________________ Edson Luiz Riccio, Professor Livre Docente da FEA USP, diretor do TECSI/FEA/USP e coordenador do 11º CONTECSI. Av. Prof. Luciano Gualberto, 908 FEA 3 São Paulo / SP / 05508-900 / E-mail: elriccio@usp.br Marici Cristine Gramacho Sakata, Mestre e Doutora em Ciência da Comunicação pela ECA USP. Pesquisadora do TECSI/FEA/USP / E-mail: mcsakata@usp.br Nelma Terezinha Zubek Valente, Mestre em Controladoria e Contabilidade pela FEA USP e Doutora em Ciência da Informação pela ECA USP. Pesquisadora do TECSI/FEA/USP. Professora da UEPG, Universidade Estadual de Ponta Grossa, Paraná, Brasil / E-mail: nzubek@usp.br Ligia Capobianco, Mestre em Ciência da Comunicação e pesquisadora do TECSI FEA USP, Universidade de São Paulo, São Paulo, Brasil / E-mail: ligiacapobianco@gmail.com ISSN online: 1807-1775 Publicado por/Published by: TECSI FEA USP – 2014

462

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

RESUMO Procuramos relatar neste documento as estatísticas, os dados e a importância do 11º CONTECSI – International Conference on Information Systems and Technology Management - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação, realizado nos dias 28, 29 e 30 de Maio de 2014 pelo TECSI/EAC/FEA/USP na Universidade de São Paulo, Brazil. Este relatório apresenta: Estatísticas do 11º CONTECSI, Justificativas e Objetivos do evento, Programa, Sessões Plenárias, Consórcio Doutoral e Master Colloquium, Sessões paralelas de apresentação de trabalhos, Menção Honrosa e Comitês. Salientamos a grande importância do auxílio financeiro recebido da CAPES, CNPq, FAPESP e FEA USP e o apoio da ANPAD, AIS, ISACA, UNINOVE, Universidade do Porto, Rutgers School/USA, São Paulo Convention Bureau, e CCINT-FEA-USP. Palavras-Chave: CONTECSI; Relatório; Congresso Internacional; Gestão de Tecnologia; Sistemas de Informação; Observatório.

ESTATÍSTICAS DO 11º CONTECSI: Tabela comparativa – 1º, 2º, 3º, 4º, 5º, 6º, 7º, 8º, 9º, 10º e 11º CONTECSI

Trabalhos Recebidos Total de Trabalhos Aceitos Em Sessão Paralela, Fórum de Pesquisa, Consórcio Doutoral, Comunicações e Poster Participantes Sessões Paralelas/Forum Estados Brasileiros Representados Países Representados

1º 100 90

2º 146 129

3º 210 177

4º 310 219

5º 315 246

6º 265 199

7º 370 233

8º 9º 10º 11° 331 354 350 320 185 256* 238* 263*

130 24

170 29

232 42

274 42

309 42

360 47

350 47 19

360 44 18

380 47 22

361 49 17

370 33 22

* Nesta edição, foram aceitos trabalhos para Master Colloquium e Sessão de Posters 1º

Palestrantes Internacionais Apoio de Agência de Fomento Apoio de Instituições e Associações Profissionais

2º

BNDE S CNPQ

3º 19 (*)

4º 4

5º 7

6º

7º

8º

9º

10º

11°

10 (**)

5 (***)

9 (****)

7 (+)

10 (++)

12 (+++)

CAPES CNPq FAPESP

CAPES FAPESP

CAPES FAPESP CNPq

BNDE S CAPES CAPES CAPES CAPES CAPE FAPESP FAPESP CNPq FAPESP S

* Apoio/Parceria da UNINOVE (*) Inclui palestrantes do evento conjunto internacional 11th World Continuous Auditing Conference. (**) Inclui palestrantes do evento conjunto internacional 18th World Symposium of Continuous Auditing Systems on Financial Institutions. (***) Inclui palestrantes do evento conjunto internacional 20th World Symposium of Continuous Auditing Systems on Financial Institutions. (****) Inclui palestrantes do evento conjunto internacional 22th World Symposium of Continuous Auditing Systems on Financial Institutions. (+) Inclui palestrantes do evento conjunto internacional 25th World Symposium of Continuous Auditing and Reporting Systems Symposium

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

463

(++) Inclui palestrantes do evento conjunto internacional 27th World Symposium of Continuous Auditing and Reporting Systems Symposium (++) Inclui palestrantes do evento conjunto internacional 31st World Symposium of Continuous Auditing and Reporting Systems Symposium

BRAZIL, AL 3

4 3

BRAZIL, DF

19 3

BRAZIL, ES

13 1

BRAZIL, GO

BRAZIL, MA

BRAZIL, MG

BRAZIL, MS

4 4

2 4

62 1

BRAZIL, PA

BRAZIL, PB

BRAZIL, PE

3 5

BRAZIL, PI

3 1

BRAZIL, PR

BRAZIL, RJ

BRAZIL, RO

BRAZIL, SC

6 6

20 1

2 6

BRAZIL, TO

13 2

BRAZIL, SE BRAZIL, SP

3 4

2 4

BRAZIL, RS

2 5

BRAZIL, RN

3 22

7 3 3

1 7

6 17

268 1

BULGARIA

COLOMBIA

MEXICO PORTUGAL

TOTAL

POSTER

POST

MSC

BRAZIL, CE

ITALY

DOCT

BRAZIL, BA

CANADA

COMM

XBRL

SOC

SEC

PRM

KMG

ITM

ISM

INV

INT

ICT

HEA

ESD

EGOV

EDU

ECOM

AUD

AIS

Speaker

Country/ País

Authors by origin and research topic - Autores por origem e temática

7 2

SPAIN

UNITED STATES

TOTAL

2 8

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

655

464

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

A) JUSTIFICATIVAS E OBJETIVOS

O 11º CONTECSI International Conference on Information Systems and Technology Management dá continuidade a um dos primeiros eventos desta área, focados na Gestão da Tecnologia e Sistemas de Informação sob uma visão multidisciplinar. O CONTECSI tem reunido, com sucesso, acadêmicos e profissionais envolvidos com a temática de gestão da tecnologia e sistemas de informação para discussão do estado da arte deste campo. O 11º CONTECSI contou com a presença de palestrantes de renome, tendo, nesta edição, um total de 263 trabalhos apresentados em 53 sessões diferentes, como sessão paralela, fórum de pesquisa, pôster, comunicação de pesquisa, consórcio doutoral, mestral e pós-doutoral onde se discutiram os efeitos da Tecnologia e dos Sistemas de Informação na Sociedade e nas Organizações. Seu principal objetivo foi promover o relacionamento entre as diversas comunidades envolvidas: a que produz, a que implementa, a que utiliza, a que regulamenta e a que pesquisa o tema em questão. A presença de renomados palestrantes e pesquisadores nacionais e internacionais permitiu uma integração entre a comunidade acadêmica e profissional, ampliando o interesse na pesquisa, no compartilhamento de informações atualizadas e das práticas utilizadas. O Congresso foi aberto pelo coordenador do projeto, Prof. Dr. Edson Luiz Riccio. Os detalhes sobre o desenvolvimento do evento cujas atividades aconteceram nas instalações da FEA 1 e FEA 5 localizadas na FEA-USP, Cidade Universitária, Campus da Capital, São Paulo/SP, Brasil, podem ser acompanhados por meio do Programa do evento apresentado a seguir. PROGRAM | PROGRAMA FEA 1 – Main Building FEA 5 – Auditorium Time / Hora 9h00– 17h00 10h30-10h45

WEDNESDAY May 28th | QUARTA-FEIRA 28 de Maio Registration | Credenciamento Opening Ceremony | Cerimônia de Abertura Welcome | Saudações de Boas Vindas Mestre de Cerimônia – Sr. Renato Campaña

Building/Roo m FEA 1 FEA 5

FEA Director: Prof. Dr. Reinaldo Guerreiro Chair: Prof. Dr. Edson Luiz Riccio Performance | Apresentação musical 10h45-11h45

Keynote Speaker DESIGN PRINCIPLES FOR PUBLIC SAFETY NETWORKS Profa. Dra. Jane Fedorowicz, Bentley University President of AIS – Association for Information Systems

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

FEA 5

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

465

Chair: Prof. Dr. João Alvaro Carvalho, Minho University, Portugal Chair: Prof. Dr. Edson Luiz Riccio, FEA USP Speaker: Prof. Dr. João Alvaro Carvalho, Minho University, Portugal 11h45-12h45

LAUNCH OF THE BRAZILIAN ASSOCIATION FOR INFORMATION SYSTEMS AND INICIAL STEPS OF THE BRAZILIAN CHAPTER OF AIS (Lançamento da Associação Brasileira de Sistemas de Informação e anúncio do Capítulo Brasileiro da AIS)

FEA 5

Prof. Dr. Edson Luiz Riccio FEA USP Profa. Dra. Jane Fedorowicz, Bentley University, President at AIS – Association for Information Systems Prof. Dr. João Alvaro Carvalho, Minho University, Portugal FEA 1 – 1st Floor

12h45-13h45

Lunch - Almoço

13h45-15h45

Paralell Sessions, Research Forum, Poster, Master Colloquium SESSION 1A – AIS RF Accounting and Enterprise Information Systems / Sistemas de Informações Contábeis e Empresariais Chair | Moderador: EVA MARTHA CHAPARRO SALINAS SESSION 1B - INT Internet and Virtual Communities and Social Networks / Internet e Comunidades Virtuais e Redes Sociais Chair|Moderator: MAXIMILIANO OSÓRIO DE VARGAS SESSION 1C – EDU IS and IT Education and Curriculum Development/Educação e Curriculo em SI e TI Chair|Moderator: SÉRGIO ANTÔNIO PULZI JÚNIOR SESSION 1D – EGOV E-governance and Public Policies / Governo Eletronico e Políticas Públicas Chair|Moderator: MEHRAN MISAGHI SESSION 1E – ESD Engineering and Software Development / Engenharia e Desenvolvimento de Software Chair|Moderator: WELLINGTON SOUSA AGUIAR

SESSION 1F – ISM Information Systems Management / Gestão de Sistemas de Informação Chair|Moderator: ADRIANA VIEIRA DA SILVA PISSINATO

FEA 1 – ROOM| SALA 6

SESSION 1G – INV IS and IT Innovation and Change / Inovações e mudanças em SI e TI Chair|Moderator: ADALBERTO FISCHMANN

FEA 1 – ROOM| SALA 7

SESSION 1H – KGM A Knowledge Management and Business Intelligence / Gestão do Conhecimento e BI Chair|Moderator EDUARDO VICENTE DE SÁ

FEA 1 – ROOM| SALA 8

SESSION 1I – ITM ENG Information Technology Management/ Gestão de Tecnologia de Informação Chair|Moderator RAFAEL TIMÓTEO DE SOUSA JUNIOR

FEA 1 – ROOM| SALA 9

PAINEL: A popularização da Ciência, Profa. Dra. Ana Nélo, Prof. Dr. Antonio Oliveira, Vice- Reitor, UFMA.

FEA 1 – ROOM| SALA 10 FEA 1 – ROOM| SALA 10

SESSION 1J – POSTER ROOM|SALA 9 Chair|Moderator: DENIS ALVES LIMA e ALANDEY SEVERO LEITE DA SILVA

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

466

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

15h45-16h00

Coffee Break

16h00-17h00

MASTER AND DOCTORAL COLLOQUIUM Chair: Prof. Dr. George Leal Jamil Prof. Dr. César Biancolino Prof. Dr. Alessandro Rosini

- FEA 1 1st. Floor FEA 1 Congregation Hall

SESSION 2A – AIS-PS I Accounting and Enterprise Information Systems / Sistemas de Informações Contábeis e Empresariais Chair|Moderador: YGOR FREITAS DE ALMEIDA

FEA 1 – ROOM| SALA 1

SESSION 2B – AUD Systems Auditing and IT Governance / Auditoria de Sistemas e Governança em TI Chair|Moderador: EDSON SAMPAIO DE LIMA

FEA 1 – ROOM| SALA 2

SESSION 2C – INVIII IS and IT Innovation and Change / Inovações e mudanças em SI e TI Chair|Moderador: MÁRCIA RODRIGUES CAPELLARI

FEA 1 – ROOM| SALA 3

SESSION 2D – ISM II Information Systems Management / Gestão de Sistemas de Informação Chair|Moderador: RAQUEL MUNIZ

FEA 1 – ROOM| SALA 4

SESSION 2E – ITM RF Information Technology Management/ Gestão de Tecnologia de Informação Chair|Moderador: ADICINÉIA A. DE OLIVEIRA

FEA 1 – ROOM| SALA 5

SESSION 2F – HEA Healthcare Information Systems and Technology/ Sistemas de Informação em Saúde e Tecnologia Chair|Moderador: EMERSON ANTONIO MACCARI

FEA 1 – ROOM| SALA 6

SESSION 2G ITM Information Technology Management/ Gestão de Tecnologia de Informação Chair|Moderador: ANTONIO JOSÉ BALLONI

FEA 1 – ROOM| SALA 7

SESSION 2H – PRM Project Management / Gestão de Projetos Chair|Moderador: ADRIANE MARIA ARANTES DE CARVALHO

FEA 1 – ROOM| SALA 8 FEA 1 – ROOM| SALA 9 FEA 1 – ROOM| SALA 10

SESSION 2I POSTER Chair|Moderator DENIS ALVES LIMA SESSION 2J COMM I Research Communication / Comunicação de Pesquisa Chair|Moderator ALEXANDRE BORBA SALVADOR 17h00-18h30

FEA 1 – ROOM| SALA 11

WORKSHOP A missão da Tecnologia, Prof. Dr. Valdemar W. Setzer MSC – Master Colloquium

May 28th 28 de Maio

14h00 - 17h

SOBRECARGA DO USO DE TECNOLOGIA MÓVEL: PROPOSTA DE ELEMENTOS PARA ESTUDO COM PROFISSIONAIS DO MERCADO FINANCEIRO - Carla Marcolin AVALIAÇÃO

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

MATURIDADE

www.jistem.fea.usp.br

PROCESSOS

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

467

TECNOLÓGICOS: SURVEY EM INSTITUIÇÕES BANCÁRIAS - Elson Luciano Weber DECISÃO GERENCIAL INSTANTÂNEA EM CONTEXTO DE UBIQUIDADE TECNOLÓGICA: UM ESTUDO COM GESTORES DO AGRONEGÓCIO - Letícia Gomes dos Santos BUSINESS INTELLIGENCE NAS PEQUENAS E MÉDIAS EMPRESAS - Alexandre da Silva Pereira ESTRUTURA ANALÍTICA DE RISCOS EM PROJETOS DE DESENVOLVIMENTO DE SOFTWARE NO SETOR BANCÁRIO - Marco Alexandre Terlizzi

May 28th 28 de Maio

Time / Hora 09h00-10h30

9h00 -10h15

10h30-10h45 10h45am 6h30pm 11h00-12h00

17h00-18h30

Workshop A MISSÃO DA TECNOLOGIA - Valdemar W. Setzer

THURSDAY May 29th | QUINTA-FEIRA 29 de Maio 2nd INTERNATIONAL PANEL ON INNOVATION IN GOVERNMENT HOW SMART SHOULD SMART CITIES BE? Panelists: Alfredo Terzoli, Rhodes University, South Africa, Smart Communities for Smart Development; Elena Ferrari, University of Insubria, Italy, Security and Privacy Concerns in Smart Cities; Joao Luiz Afonso, University of Minho, Portugal Isabel Ramos, University of Minho, Portugal José Antonio Gomes de Pinho – NPGA - Escola de Administração UFBA, Brazil, João Alvaro Carvalho, University of Minho, Portugal Chairs: Prof. Dr. Walter Castelnovo, University of Insubria, Italy Prof. Dr. Edson Luiz Riccio, University of São Paulo, Brazil

Building/Roo m FEA 5

DOCTORAL CONSORTIUM, MASTER COLLOQUIM AND POST DOCTORAL CONSORTIUM (FROM 9H TO 17H00 SEE PROGRAM BELOW) SESSION– DOCT I Doctoral Consortium Chair: Prof. Dr. George Leal Jamil Prof. Dr. César Biancolino Prof. Dr. Alessandro Rosini

CONGREGA TION HALL | SAL A DA CONGREGA ÇÃO

Coffee Break 31st WCARS World Continuous Auditing and Reporting Systems Symposium – See program below PUBLISHING YOUR RESEARCH PAPER Prof. Dr. João Alvaro Carvalo, Minho University, Portugal

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

FEA 1 - 1. Floor / FEA 5 CONGREGA TION

468

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L. Profa. Dra. Jane Fedorowicz, Bentley University, USA Prof. Dr. Jean-Claude Guedon, Montreal University, Canada

12h00-13h00 13h45-15h45

Lunch - FEA 1 - 1st Floor | Almoço - FEA 1 - 1. Andar SESSION 3A – AIS - PS II Accounting and Enterprise Information Systems / Sistemas de Informações Contábeis e Empresariais Chair|Moderator: MAURÍCIO LEITE

FEA 1 – ROOM| SALA 2

SESSION 3D – EDU IS and IT Education and Curriculum Development/Educação e Currículo em SI e TI Chair|Moderator: EDISON LUIZ GONÇALVES FONTES

FEA 1 – ROOM| SALA 3

SESSION 3E – EGOV II E-governance and Public Policies / Governo Eletrônico e Políticas Públicas Chair|Moderator: ERNANI MARQUES DOS SANTOS

FEA 1 – ROOM| SALA 4

SESSION 3F – ESD RF I Engineering and Software Development / Engenharia e Desenvolvimento de Software Chair|Moderator: EDILAINE SOARES

FEA 1 – ROOM| SALA 5

SESSION 3G – INV II IS and IT Innovation and Change / Inovações e mudanças em SI e TI Chair|Moderator: ADEMIR MACEDO NASCIMENTO

FEA 1 – ROOM| SALA 6

SESSION – DOCT I Doctoral Consortium Prof. Dr. George Leal Jamil Prof. Dr. César Biancolino

15h45-16h00

FEA 1 – ROOM| SALA 7 Congregation Hall | Sala da Congregação

SESSION 4A – AIS - PSIII Accounting and Enterprise Information Systems / Sistemas de Informações Contábeis e Empresariais Chair|Moderador: ADRIANO DINOMAR BARP

FEA 1 - 1st Floor | FEA 1 - 1. Andar FEA 1 – ROOM| SALA 1

SESSION 4B – SOC Social Issues in IS and IT / Questões sociais em SI e TI Chair|Moderator: JULIO ALVAREZ BOTELLO

FEA 1 – ROOM| SALA 2

SESSION 4C – SEC IS and IT Security / Segurança em SI e TI Chair|Moderator: FRANCISCO GILVAN LIMA MOREIRA

FEA 1 – ROOM| SALA 3 FEA 1 –

Coffee Break 16h00-18h00

FEA 1 – ROOM| SALA 1

SESSION 3B*– XBRL XBRL and Enterprise Onthology / XBRL e Ontologia empresarial Chair|Moderator: ALANDEY SEVERO LEITE DA SILVA *Novo horário: 16h às 18h SESSION 3C – ICT-RF ICT4DEV – Information and Communication Technologies for Development and Assistive Technologies / TICS para Desenvolvimento Chair|Moderator: ALEXANDRE MORAIS DE SOUZA

SESSION 3H – PRM II Project Management / Gestão de Projetos Chair|Moderator: MICHEL LENS SELLER 13h45-15h45

HALL | SAL A DA CONGREGA ÇÃO

SESSION 4D – INT-RF Internet and Virtual Communities and Social

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

16h00-17h00

469

Networks / Internet e Comunidades Virtuais e Redes Sociais Chair|Moderator: ANA MARIA NÉLO

ROOM| SALA 4

SESSION 4E – EDU-RF IS and IT Education and Curriculum Development/Educação e Currículo em SI e TI Chair|Moderator: MINERVA MARTÍNEZ ÁVILA

FEA 1 – ROOM| SALA 5

SESSION 4F – HEA II Healthcare Information Systems and Technology/ Sistemas de Informação em Saúde e Tecnologia Chair|Moderator: LILIANA GIUSTI SERRA

FEA 1 – ROOM| SALA 6

SESSION 4G – ITM II Information Technology Management/ Gestão de Tecnologia de Informação Chair|Moderator: ALBERTO MEDEIROS JR.

FEA 1 – ROOM| SALA 7

SESSION 4H – COMM II Research Communication / Comunicação de Pesquisa Chair|Moderator: CECÍLIA MORAES SANTOSTASO GERON

FEA 1 – ROOM| SALA 8

SESSION– POST I Postdoctoral Consortium Congregation Hall | Sala da Congregação Prof. Dr. George Leal Jamil Prof. Dr. César Biancolino Prof. Dr. Armando Malheiro

17h00-18h30

Congregation Hall | Sala da Congregação

WORKSHOP INTELIGÊNCIA DE MERCADO - George Leal Jamil 19h00-22h00

OFFICIAL DINNER | JANTAR DE CONFRATERNIZAÇÃO Bus departure from USP 19h00 return at 22h30 Saída de ônibus da USP as 19h00 e retorno às 22h30

Golf São Francisco

DOCT - Doctoral Consortium / Consórcio Doutoral GESTÃO DE CONHECIMENTO NO ALINHAMENTO ENTRE TECNOLOGIA DA INFORMAÇÃO E ESTRATÉGIA EMPRESARIAL NA BUSCA DE DIFERENCIAL COMPETITIVO - Claúdio Roberto Magalhães Pessoa

May 29th 29th de Maio

9h00 – 10h30

PROCESSO DE CRIAÇÃO DO CONHECIMENTO EM SISTEMAS ADAPTATIVOS COMPLEXOS - Diego Gilberto Ferber Pineyrua CULTURA ORGANIZACIONAL: VALORES E REDES Ines Francisca Neves Silva DOCT - Doctoral Consortium / Consórcio Doutoral

10h30 - 10h45

IMPLICAÇÕES DAS ALIANÇAS ESTRATÉGICAS INTRAFIRMAS NA TRANSFERÊNCIA DE ATIVIDADES DE DESENVOLVIMENTO DE PRODUTOS - Evandro Prieto Coffee-break

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

470

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L. Painel Publishing your Research Paper

11h00-12h00

Lunch/Almoço

12h00 - 13h00

DOCT - Doctoral Consortium / Consórcio Doutoral DIAGNÓSTICO E ANÁLISE DA ESTRUTURA DO ATENDIMENTO À SAÚDE DOS AGENTES DE SEGURANÇA PÚBLICA: ESTUDO NA SECRETARIA DE ESTADO DE SEGURANÇA PÚBLICA DO ESTADO DE MATO GROSSO (SESP- MT) CONSTRUÍDO COM BASE NA TEORIA DE REDES E TEORIA DE CAMPOS Laudicerio Aguiar Machado 13h45 -15h45

15h45 - 16h00

O PAPEL DO AMBIENTE VIRTUAL DE APRENDIZAGEM NA GESTÃO DO PROJETO ALUNO INTEGRADO DA UFSC: UMA TEORIA SUBSTANTIVA - Paulo Cristiano de Oliveira SE QUER QUE SEJA BEM FEITO, FAÇA EM EQUIPE: UMA INVESTIGAÇÃO SOBRE ANTECEDENTES E CONSEQUENTES DO ESTADO DE FLUXO EM EQUIPES DE TECNOLOGIA DA INFORMAÇÃO - Pedro Jacome de Moura Junior Coffee-break POST – Post doctoral Consortium/ Consórcio Pós-Doutoral

16h00 – 17h00

MODELO VIVENCIAL DE APRENDIZAGEM INTERCULTURAL EM SIMULADORES DE EMPRESAS Luiz Antonio Titton

Workshops 17h00 - 18h30

Time / Hora 09h00-10h30

9h00 -11h00

10h30-10h45 10h45-12h00

INTELIGÊNCIA DE MERCADO – Prof. George Leal Jami

FRIDAY May 30th | SEXTA-FEIRA 30 de Maio INFORMATION SCIENCE PANEL Prof. Dr. Jean-Claude Guédon, Montreal University, Canada Chair: Prof. Dr. Armando Malheiro, University of Porto, Portugal Discussant: Andre Serradas, SIBI USP

Building/Room

SESSION– MSC I Master Colloquium Prof. Dr. George Leal Jamil Prof. Dr. Cesár Biancolino Prof. Dr. João Álvaro de Carvalho Coffee break

Congregation Hall | Sala da Congregação

INFORMATION SCIENCE PANEL Prof. Dr. Luc Quoniam, Aix Marseille University, France Prof. Dr. Armando Malheiro, University of Porto, Portugal

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

FEA 5

FEA 1 - 1st Floor | FEA 5

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

11h30-12h45

12h45-13h45 13h45-15h45

471

SESSION 5A – AIS-RF II Accounting and Enterprise Information Systems / Sistemas de Informações Contábeis e Empresariais Chair|Moderator: VILMA GENI SLOMSKI

FEA 1 – ROOM| SALA 1

SESSION 5B – ECOM E-business and E-commerce / Comércio Eletrônico Chair|Moderator: MARIA IRENE DA FONSECA E SÁ

FEA 1 – ROOM| SALA 2

SESSION 5C – EDU-RF II IS and IT Education and Curriculum Development/Educação e Curriculo em SI e TI Chair|Moderator: MARCOS RICARDO ROSA GEORGES

FEA 1 – ROOM| SALA 3

SESSION 5D – EGOVIII E-governance and Public Policies / Governo Eletrônico e Políticas Públicas Chair|Moderator: CLAUDIA MARIA DA COSTA ARCHER

FEA 1 – ROOM| SALA 4

SESSION 5E – ISM-RF Information Systems Management / Gestão de Sistemas de Informação Chair|Moderator: RENATO RIBEIRO NOGUEIRA FERRAZ

FEA 1 – ROOM| SALA 5

SESSION 5F – ITM RF II Information Technology Management/ Gestão de Tecnologia de Informação Chair|Moderator: GILBERTO PEREZ

FEA 1 – ROOM| SALA 6

SESSION 5G – ITM III Information Technology Management/ Gestão de Tecnologia de Informação Chair|Moderator: JOÃO CARLOS HIPÓLITO

FEA 1 – ROOM| SALA 7

Reunião Geral da Associação Brasileira de Sistemas de Informação e organização do Capítulo Brasileiro da AIS Lunch| Almoço SESSION 6A – AIS -PS IV Accounting and Enterprise Information Systems / Sistemas de Informações Contábeis e Empresariais Chair|Moderator: RODRIGO BARRACO MARASSI

Congregation Hall | Sala da Congregação FEA 1 - 1st Floor FEA 1 – ROOM| SALA 1

SESSION 6B – ICT ICT4DEV – Information and Communication Technologies for Development and Assistive Technologies / TICS para Desenvolvimento PP Chair|Moderator: ALMIR CARLOS DA SILVA

FEA 1 – ROOM| SALA 2

SESSION 6C – ICT-RF II ICT4DEV – Information and Communication Technologies for Development and Assistive Technologies / TICS para Desenvolvimento Chair|Moderator RAFAEL HERDEN CAMPOS

FEA 1 – ROOM| SALA 3

SESSION 6D – ESD RF II Engineering and Software Development / Engenharia e Desenvolvimento de Software Chair|Moderator: MARCO ALEXANDRE TERLIZZI

FEA 1 – ROOM| SALA 4

SESSION 6E – HEARF - ENG Healthcare Information Systems and Technology / Sistemas de Informação em Saúde e Tecnologia Chair|Moderator ALESSANDRO MARCO ROSINI

FEA 1 – ROOM| SALA 5

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

472

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L. SESSION 6F – ISM III Information Systems Management / Gestão de Sistemas de Informação Chair|Moderator: JAIRO SIMIÃO DORNELAS

FEA 1 – ROOM| SALA 6

SESSION 6G – KGM II Knowledge Management and Business Intelligence / Gestão do Conhecimento e BI Chair|Moderator: ITIEL MORAES DA SILVA

FEA 1 – ROOM| SALA 7

SESSION 6H – PRM RF Project Management / Gestão de Projetos Chair|Moderator: ÉRICA SOUZA SIQUEIRA

FEA 1 – ROOM| SALA 8 FEA 1 – ROOM| SALA 9

SESSION 6I – COMMIII Research Communication Comunicação de Pesquisa Chair|Moderator: DENIS BRUNO VIRÍSSIMO

15h45-16h00

Coffee break

16h00-17h00

BEST PAPER AWARD & CLOSING CEREMONY | MELHORES TRABALHOS DO 11º CONTECSI E ENCERRAMENTO

FEA 1 - 1st Floor | FEA 5

Musical Performance - Apresentação Musical

MSC – Master Colloquium

May 30th 30 de Maio

ANTECEDENTES DA SATISFAÇÃO NO ENSINO A DISTÂNCIA-EAD: ESTUDO BASEADO NA TEORIA DO SUCESSO EM SISTEMAS DE INFORMAÇÃO (ISS) E NA TEORIA DA PRÉ-DISPOSIÇÃO A TECNOLOGIA (TRI) Marcia Welita da Silva 9:00h – 11:00h

CONTROLES INTERNOS E DE TECNOLOGIA DA INFORMAÇÃO NA MITIGAÇÃO DOS RISCOS DE CONFORMIDADE DAS INFORMAÇÕES CONTÁBEIS Maria Cecília da Silva Brum O USO DO PLANEJAMENTO DE TI: ESTUDO DE CASO EM EMPRESAS DO RIO DE JANEIRO - Leonardo Porto Carvalho

B) SESSÕES PLENÁRIAS: DESIGN PRINCIPLES FOR PUBLIC SAFETY NETWORKS Jane Fedorowicz, PhD President, Association for Information Systems Chester B. Slade Professor of Accounting and Information Systems Bentley University Waltham, MA USA

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

473

Public safety organizations are limited in their ability to communicate and share information with other agencies even though they have the technology in place to do so within their own boundaries. The goal of this research is to help agencies realize the value of joining together to design, develop and deploy information and communications technologies to support policing, criminal justice, public safety and homeland security. This research pursues two main questions about cross-agency communications and information sharing: • Why do public safety collaborations occur in some places and not others? • What makes public safety collaboration successful? To answer these questions, the project team has developed a U.S.-based, state-by-state data set of public safety networks. We identified over 275 initiatives at the state and local levels and have collected detailed interview data on 80 of these. We have also gathered data on over 230 independent variables for each of the 50 states in the U.S., which include political, demographic, public safety, police/justice, and computing issues. Finally, we have conducted six extensive case studies. Theoretical contributions to date include the identification of the major forms of public safety networks, and contrasting theoretical explanations regarding design decisions for both collaboration infrastructure and governance, focusing on institutional and rational choice theories. The project also contributes to knowledge on the applicability and use of complex adaptive systems theory to study system implementation, and extends a common shared service framework to include the role of public policy in public sector shared service implementations. The presentation will summarize the theory, data collection and analyses that resulted in these contributions. Finally, I will present a set of sociotechnical design observations and design principles that will be useful for many public sector systems that cross organizational boundaries. OPEN ACCESS: GLOBALIZING SCIENCE IN THE RIGHT WAY Prof. Dr. Jean-Claude Guédon Comparative Literature Department University of Montreal, Canada Open access holds the promise of offering more than access. In the case of countries such as Brazil, it also provides the possibility of influencing the globalization of science in very positive ways. In particular, it can help redefine criteria of authority and prestige in such a way that researchers from Brazil and elsewhere can pursue problems of local interest, publish in good journals that are not necessarily those selected by the Web of Science or Scopus, and yet attain a level of visibility that can serve their career ambitions and research programmes. Open access, in short, provides the means to redefine scientific value, as well as the means to redefine the parameters of scientific competition so that it does not lead to a situation where scientists from emerging nations need to work on problem areas dictated elsewhere. Open access can also help establish other value criteria beside “excellence” - itself the result of particular forms of competition - to reassert the primacy of quality as the most fundamental dimension of research.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

474

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

A CIÊNCIA DA INFORMAÇÃO NA INFOESFERA: ESTRATÉGIAS E DESAFIOS Prof. Dr. Armando de Barros Malheiro Faculdade de Letras da Universidade do Porto Como tem sido regra, também a Mesa Plenária em Ciência da Informação do 11º CONTECSI foi temática e subordinada ao título: A Ciência da Informação na Infoesfera: estratégias e desafios. A discussão do tema teve como propósito aprofundar, nas várias dimensões possíveis, a relação estreita que as abordagens passíveis de serem desenvolvidas no seio da Ciência da Informação permitem para uma compreensão científica da Infoesfera. A Ciência da Informação não é uma sociologia ou uma ciência social pura, mas assomadamente uma ciência social aplicada que precisa identificar problemas e fenômenos carecidos de explicação, tendo em vista a busca de novas condições concretas mais ajustadas e facilitadoras. A Infoesfera é metáfora para a realidade cada vez mais complexa em que nos situamos, na qual estamos cada vez mais mergulhados, enquanto observamos que o impacto da revolução informacional em nossas vidas é incontornável. HOW SMART SHOULD SMART CITIES BE? SECOND INTERNATIONAL PANEL ON INNOVATION IN GOVERNMENT Panelists Pallab Saha, National University of Singapore, Why Smart City Initiatives Can Fail Alfredo Terzoli, Rhodes University, South Africa, Smart Communities for Smart Development Elena Ferrari, University of Insubria, Italy, Security and Privacy Concerns in Smart Cities José Antonio Gomes de Pinho - NPGA-Escola de Administração UFBA, Brazil João Alvaro Carvalho, University of Minho, Portugal Chairs Walter Castelnovo, University of Insubria, Italy Edson Luiz Riccio, University of Sao Paulo, Brazil Smart cities (and smart communities) are rapidly becoming one of the hottest topics in discussions about innovation. There is a continuously growing interest on smart cities and smart communities from ICT companies, central and local governments, academic communities and, more generally, civil society. However, it is not yet clear at all whether this global trend toward smart cities and smart communities will actually lead to achieve the stated objectives, which are far reaching and challenging. These objectives concern all the dimensions that characterize smart cities (i.e. smart economy, smart people, smart governance, smart mobility, smart environment and smart living) and require a smart use of ICTs. This makes the implementation of smart cities/communities initiatives a very complex and highly resource consuming process. Even when they are implemented through public-private partnerships, the funding of smart cities/communities initiatives mainly, if not exclusively, depends on public resources. From this point of view, especially in times of public expenditure cuts,

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

475

learning from past experiences (both from successes and from failures) is crucial in order to avoid wasting public resources. This also means carefully considering the problems that could arise in and from the implementation of smart cities/communities initiatives, besides stressing the benefits that could possibly derive from them. The objective of the panel is to discuss from different points of view, and with respect to different cultural and economic environments, some critical aspects of the smart cities/communities paradigm, with the aim of contributing to a critical debate that looks beyond the smart city hype. WORKSHOP INTELIGÊNCIA DE MERCADO George Leal Jamil A inteligência de mercado é conceituada como um processo empresarial que visa prover continuamente conhecimentos para o planejamento estratégico de marketing. Neste workshop o conceito foi aplicado na análise de casos reais e um roteiro de realização de ciclos de inteligência de mercado foi desenvolvido, para fins de estudos acadêmicos e uso por setores empresariais competitivos. C) DOCTORAL CONSORTIUM E MASTERS COLLOQUIUM DOCT - Doctoral Consortium / Consórcio Doutoral GESTÃO DE CONHECIMENTO NO ALINHAMENTO ENTRE TECNOLOGIA DA INFORMAÇÃO E ESTRATÉGIA EMPRESARIAL NA BUSCA DE DIFERENCIAL COMPETITIVO - Claúdio Roberto Magalhães Pessoa PROCESSO DE CRIAÇÃO DO CONHECIMENTO EM SISTEMAS ADAPTATIVOS COMPLEXOS - Diego Gilberto Ferber Pineyrua CULTURA ORGANIZACIONAL: VALORES E REDES - Ines Francisca Neves Silva IMPLICAÇÕES DAS ALIANÇAS ESTRATÉGICAS INTRAFIRMAS NA TRANSFERÊNCIA DE ATIVIDADES DE DESENVOLVIMENTO DE PRODUTOS Evandro Prieto DIAGNÓSTICO E ANÁLISE DA ESTRUTURA DO ATENDIMENTO À SAÚDE DOS AGENTES DE SEGURANÇA PÚBLICA: ESTUDO NA SECRETARIA DE ESTADO DE SEGURANÇA PÚBLICA DO ESTADO DE MATO GROSSO (SESP- MT) CONSTRUÍDO COM BASE NA TEORIA DE REDES E TEORIA DE CAMPOS - Laudicerio Aguiar Machado O PAPEL DO AMBIENTE VIRTUAL DE APRENDIZAGEM NA GESTÃO DO PROJETO ALUNO INTEGRADO DA UFSC: UMA TEORIA SUBSTANTIVA - Paulo Cristiano de Oliveira SE QUER QUE SEJA BEM FEITO, FAÇA EM EQUIPE: UMA INVESTIGAÇÃO SOBRE ANTECEDENTES E CONSEQUENTES DO ESTADO DE FLUXO EM EQUIPES DE TECNOLOGIA DA INFORMAÇÃO - Pedro Jacome de Moura POST – Post doctoral Consortium/ Consórcio Pós-Doutoral MODELO VIVENCIAL DE APRENDIZAGEM INTERCULTURAL EM SIMULADORES

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

476

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

DE EMPRESAS - Luiz Antonio Titton OBSERVATÓRIO CONTECSI – CRIAÇÃO E IMPLEMENTAÇÃO: ANÁLISE DA DINÂMICA CIENTÍFICA ALCANÇADA E MONITORAMENTO DO IMPACTO DO EVENTO EM NÍVEL NACIONAL E INTERNACIONAL – Marici Gramacho Sakata MSC – Master Colloquium ANTECEDENTES DA SATISFAÇÃO NO ENSINO A DISTÂNCIA - EAD: ESTUDO BASEADO NA TEORIA DO SUCESSO EM SISTEMAS DE INFORMAÇÃO (ISS) E NA TEORIA DA PRÉ-DISPOSIÇÃO À TECNOLOGIA (TRI) - Marcia Welita da Silva CONTROLES INTERNOS E DE TECNOLOGIA DA INFORMAÇÃO NA MITIGAÇÃO DOS RISCOS DE CONFORMIDADE DAS INFORMAÇÕES CONTÁBEIS - Maria Cecília da Silva Brum O USO DO PLANEJAMENTO DE TI: ESTUDO DE CASO EM EMPRESAS DO RIO DE JANEIRO - Leonardo Porto Carvalho SOBRECARGA DO USO DE TECNOLOGIA MÓVEL: PROPOSTA DE ELEMENTOS PARA ESTUDO COM PROFISSIONAIS DO MERCADO FINANCEIRO - Carla Marcolin AVALIAÇÃO DE MATURIDADE EM PROCESSOS TECNOLÓGICOS: SURVEY EM INSTITUIÇÕES BANCÁRIAS - Elson Luciano Weber DECISÃO GERENCIAL INSTANTÂNEA EM CONTEXTO DE UBIQUIDADE TECNOLÓGICA: UM ESTUDO COM GESTORES DO AGRONEGÓCIO - Letícia Gomes dos Santos BUSINESS INTELLIGENCE NAS PEQUENAS E MÉDIAS EMPRESAS - Alexandre da Silva Pereira ESTRUTURA ANALÍTICA DE RISCOS EM PROJETOS DE DESENVOLVIMENTO DE SOFTWARE NO SETOR BANCÁRIO - Marco Alexandre Terlizzi

D) 31º WORLD CONTINUOUS AUDITING AND REPORTING SYSTEMS SYMPOSIUM O 31º. Simpósio Mundial de Auditoria Contínua de Sistemas reuniu em torno de 100 pessoas no Auditório da FEA/USP, no dia 29/05/2014 e trouxe para discussão assuntos de interesses acadêmicos e profissionais, tais como: Big Data e a transformação da auditoria, certificações profissionais para auditoria, XBRL e Webfilling, mecanismos de busca e análise de dados, como a auditoria contínua pode auxiliar a lei 12.846/2013 – Anticorrupção Brasileira, além de estudos de casos reais de auditoria contínua. O Simpósio contou com a participação de palestrantes experientes no âmbito acadêmico e profissional. Como destaques, tivemos a presença dos professores Deniz Appelbaum, Dra. Helen Brown, Dr. Kevin Moffitt e Miklos Vasarhelyi da Rutgers University e do Dr. Rajendra P. Srivastava da University of Kansas, USA. Foram também apresentados casos práticos de implementação de Auditoria

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

477

Contínua em empresas brasileiras, as quais ganharam maior eficiência e produtividade em seus processos de auditoria. Como considerações finais, podemos afirmar que os métodos aplicados para o desenvolvimento da auditoria contínua contam com novas tendências tecnológicas para a exploração de dados, agregando valor aos processos de negócios das organizações e às técnicas de auditoria. 31ST WCARS WORLD CONTINUOUS AUDIT AND REPORTING SYSTEMS SYMPOSIUM O WCARS (World Continuous Audit and Reporting Systems Symposium) é um congresso que ocorre de 3 a 4 vezes ao ano ao redor do mundo. O WCARS enfatiza a automação de auditoria, métodos de auditoria analítica, e auditoria na era de tempo real que chamamos de auditoria contínua. Organizações que lideram o mundo nesta área e acadêmicos de vanguarda apresentam novas idéas para melhorar as auditorias internas e externas. O Brasil tem estado na vanguarda deste esforço, em particular nas organizações financeiras. COMO A AUDITORIA CONTÍNUA PODE AUXILIAR A LEI 12.846/2013 – ANTICORRUPÇÃO BRASILEIRA Washington Lopes Foram apresentados os principais aspectos da Lei Anticorrupção Brasileira: o que muda para as empresas nos aspectos de Compliance, Controles Internos e Auditoria Interna. Apresentaram-se também os possíveis cenários de mudanças e como as técnicas de Auditoria Contínua podem apoiar na monitoração dos controles internos relacionados aos aspectos da lei 12.846/2013. TEXT ANALYTICS IS ADVANCING AS AN ACCEPTED AND USEFUL METHOD IN ACCOUNTING RESEARCH Kevin Mofitt Text analytics is advancing as an accepted and useful method in accounting research (Li 2010a; Fisher et al. 2010). However, in the accounting practice, external auditors have yet to embrace the benefits or find a significant use for the research or analytical tools that have been developed in academia. For this project, we have developed a tool that analyzes text contained in publicly available financial documents. The purpose of the tool is to assist auditors in identifying risks associated with the company in the planning stage of the audit. The tool accomplishes this with two main functions: (1) the tool analyzes and reports linguistic trends found in the text of the documents and (2) the tool identifies new and deleted text in documents that were updated from one year to the next. We have two main hypotheses regarding the effect the tool will have on auditor performance: (H1) Auditors using the tool will assess risks more effectively than those not using the tool, and (H2) Auditors using the tool will assess risks more efficiently than those not using the tool. As research in progress, the hypotheses are still in the process of being tested. In this stage of the project I will demonstrate the tool on realworld data.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

478

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

EXPERT SYSTEMS: INTERNAL ANOMALY DETECTION IN THE AREA OF EMPLOYEE PROCUREMENT Deniz Applebaum Expert Systems: Internal Anomaly Detection in the area of employee procurement cards continues to be an issue within many firms, even large multinational corporations. In his work in progress study, the authors observed the manual procedures that were being employed by a firm and are creating detailed scripts imitating and enhancing these processes. This firm reviews over 50,000 employee procurement card transactions per month. There are over 55 attributes defined for each transaction. The data set begins in 2011 and is updated monthly. Meanwhile, the firm continued to employ its internal manual fraud detection procedures, as a control setting for this study. Based on initial preliminary results, these scripts are expected to improve the success rate of anomaly detection for this firm, plus greatly ease and accelerate its fraud detection process. This research is important as it contributes to the literature of anomaly detection in continuous auditing and monitoring. By replicating the human processes and judgment procedures on many levels, scripts can improve the audit detection rate at many firms in the review of employee procurement card purchases. AUDIT DATA STANDARDS FOR U.S. AND BRAZILIAN NON-PROFITS: Deniz Applebaum Audit Data Standards for U.S. and Brazilian non-profits: This is a joint research project between a U.S. university and a Brazilian university. The project teams will undertake an analysis of the applicability of currently defined Continuous Audit protocols in the notfor- profit (NFP) segment. As part of this process the teams will review currently applicable audit data standards (ADSs) in both the U.S. and Brazil to determine their applicability for NFP organizations and contrast differences that may exist between the two countries. In the case of Brazil, Continuous Audit protocols have not yet been established as usual practices. However, due to its usefulness and potential benefits, it is relevant to investigate to what extent Brazilian non-profits, especially the foundations linked to the government and public universities and hospitals, could benefit from Continuous Audit / Continuous Monitoring standards. This work in progress paper addresses the applicability of these ADSs in the SME NFP environment in both the US and Brazil, and develops a global vision for this process.

BIG DATA AND THE TRANSFORMATION OF AUDITING Miklos A. Vasarhelyi Over the last two decades a slow evolution towards a more “continuous” and “analyticbased” auditing has started to change the nature of auditing. Terminologies like predictive, preventive, and analytic audit have become more common as the needs of assurance on progressively complex and opaque automated systems have emerged. Now the “big data” environment has been added to the equations and new challenges and opportunities.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

479

This talk focuses on new forms of evidence and how they can be used to totally revamp auditing in the form of an assurance eco-system.

BEHAVIORAL IMPLICATIONS OF BIG DATA’S IMPACT ON AUDIT JUDGMENT AND DECISION MAKING AND FUTURE RESEARCH DIRECTIONS Helen Brown The proliferation of computerized accounting information systems and the low cost of storage have led to a dramatic increase in the size of data generated and collected by companies. While capturing Big Data is relatively easy, the process of analyzing this data poses several issues. This paper looks at the current state of Big Data with regards to the audit environment and discusses how Big Data differs from traditional data. Next, the behavioral implications that Big data has on audit judgment is presented by addressing the issues of information overload, information relevance, pattern recognition, and ambiguity. The paper draws on the psychology and auditing literature to investigate the effects of information overload that auditing Big Data introduces. Information overload, in turn, leads to the problem of identifying relevant information. Moreover, Big Data aggravates the difficulties of identifying patterns in the data, which makes it less likely for less experienced auditors to incorporate it in their judgment. In addition to that, the unstructured nature of Big Data increases ambiguity, which has been shown in the literature to affect audit efficiency. The paper also discusses the challenges that auditors encounter when dealing with Big Data and the various analytical tools that are currently used by companies in the analysis of Big Data. The last section of the manuscript concludes by raising open research questions related to auditing Big Data that future research might address. SEARCH ENGINES AND DATA ANALYTICS : THEIR RELEVANCE AND VALUE WITHIN XBRL ENVIRONMENT AND BEYOND Rajendra P. Srivastava EY Distinguished Professor and Director, EY Center for Auditing Research and Advanced Technology. University of Kansas, Lawrence, KS 66045, USA rsrivastava@ku.edu The main objective of this paper is to discuss the role of search engines and data analytics within XBRL Environment and beyond. It is frequently argued that having XBRL formatted business reports makes it easy to access information and hence analyze it. However, one major problem with the XBRL formatted data is that the analytical tools will capture only those pieces of information that are tagged. What if the user needs information that is not tagged or not required to be tagged, can the analytical tool still be effective to provide the analysis needed? Well, this is where search engines would become important. The present paper will discuss advantages and disadvantages of analytical tool built for XBRL formatted documents and demonstrate the value of search engines that go beyond the XBRL environment, especially in situation where the needed pieces of information are not being tagged. Some real world examples will be

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

480

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

discussed along with a demonstration of two search engines called “Seek iNF” and “FRAANK” developed at the University of Kansas.

E) Posters: A apresentação de posters no 11o CONTECSI teve a presença de alunos de graduação que divulgaram trabalhos resultantes de pesquisas ou desenvolvimentos realizados durante disciplinas ou projetos relacionados aos seus respectivos cursos. A quase totalidade dos trabalhos teve como objeto, a idealização de um projeto que buscasse atender a necessidades reais encontradas em diversos segmentos de negócios: pecuária, comércio, educação, esportes e vários outros. Os estudantes demonstraram segurança quanto à utilidade e aplicabilidade de seus projetos, sempre evidenciando o mercado alvo e a necessidade a ser atendida. Dentre os aspectos positivos dos trabalhos, merece destaque a desenvoltura que os alunos apresentaram no que tange à familiaridade com o desenvolvimento de sistemas, o caráter interdisciplinar de vários dos projetos, que contavam com estudantes de áreas distintas e a diversidade de ideias, variando desde projetos de equipamentos para solução de problemas de coleta seletiva de lixo até a proposta de integração de hardware e software para o auxílio no rápido atendimento a acidentes de trânsito. Dentre pontos a melhorar, pode ser destacado o desconhecimento dos alunos quanto aos procedimentos para proteção da propriedade intelectual, o que é de fundamental relevância para a sobrevivência de um empreendimento dessa natureza.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

481

31th WCARS - WORLD CONTINUOUS AUDITING AND REPORTING SYSTEMS SYMPOSIUM AUDITORIUM FEA 5 | AUDITÓRIO FEA 5

Time / Hora 10h30 – 10h45 10h45 – 10h50

10h50 – 11h05 11h05 – 11h25 11h25 – 11h50

11h50 – 12h15

12h15 – 12h45

12h45 – 13h35 13h35 – 13h55

13h55 – 14h35

14h35 – 15h20

THRUSDAY May 29th | QUINTA-FEIRA 29 de Maio PROGRAM | PROGRAMA 31st WCARS Registration and Welcome Coffee OPENNING CEREMONY Openning 31st World Continuous Auditing and Reporting Systems Symposium Prof. Dr. Edson Luiz Riccio, FEA USP/Coordenador do Comitê Técnico XBRL Brasil (CFC) Prof. Dr. Miklos Vasarhelyi, Director Rutgers Accountng Research Center & Continuous Auditing & Reporting Lab THE ELECTRONIC AUDITING AND CONTINUOUS ITAU Eduardo H. Miyaki Itau Unibanco- Director , Internal Audit BIG DATA AND THE TRANSFORMATION OF AUDITING Prof. Dr. Miklos Vasarhelyi Rutgers Business School, USA AUDIT DATA STANDARDS FOR U.S. AND BRAZILIAN NON-PROFITS Deniz Appelbaum Rutgers Business School, USA PhD student in Accounting Information Systems at Rutgers Business School Denis Lima e Alves Faculdade de Economia Administração e Contabilidade, FEA/USP BEHAVIORAL IMPLICATIONS OF BIG DATA’S IMPACT ON AUDIT JUDGMENT AND DECISION MAKING AND FUTURE RESEARCH DIRECTIONS Profa. Dra. Helen Brown Rutgers Business School, USA PROFESSIONAL CERTIFICATIONS ISACA - Information Systems Audit and Control Association Gustavo Galegale Vice Presidente do ISACA Capítulo São Paulo Lunch - FEA 1 - 1st Floor | Almoço - FEA 1 - 1. Andar EXPERT KNOWLEDGE ELICITATIONS IN A PROCUREMENT CARD CONTEXT: TOWARDS CONTINUOUS MONITORING AND ASSURANCE Deniz Appelbaum Rutgers Business School, USA PhD student in Accounting Information Systems at Rutgers Business School XBRL Palestrantes: Prof. Dr. Edson Luiz Riccio EAC/FEA USP; Coordenador do Comitê Técnico XBRL Brasil (CFC) Prof. Dr. Paulo Caetano da Silva Universidade de Salvador- Member of XBRL Certification Board e Banco Central do Brasil Profa. Cecilia Geron Praesum Contabilidade/ FEA USP Roberta Bittencourt TECSI FEA/USP SEARCH ENGINES AND DATA ANALYTICS: THEIR RELEVANCE AND VALUE WITHIN XBRL ENVIRONMENT AND BEYOND Prof. Dr. Rajendra P. Srivastava University of Kansas, USA

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

482

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

15h20 – 15h45

15h45 – 16h00 16h00 – 16h30

16h30 - 17h00 17h00 – 17h30

17h30 – 18h00

18h00 – 18h45

Liv Watson Webfilings Solutions, USA TEXT ANALYTICS IS ADVANCING AS AN ACCEPTED AND USEFUL METHOD IN ACCOUNTING RESEARCH Prof. Dr. Kevin Moffitt Rutgers Business School, USA Coffee break FEA 1 - 1st Floor | FEA 1 - 1. Andar OTIMIZANDO O PROCESSO DE AUDITORIA TRADICIONAL COM A AUDITORIA CONTÍNUA Fernando Nicolau Freitas Ferreira Sócio Diretor da AuditSafe Assessoria Empresarial ESTUDO DE CASO DA EMPRESA – VIXTEAM HOW TO IMPLEMENT CONTINUOUS AUDITING – CASE STUDIES Gustavo K. Giometti Itau Unibanco, Superintendent of Continuous Audit, Risk and Projects COMO A AUDITORIA CONTÍNUA PODE AUXILIAR A LEI 12.846/2013 – ANTICORRUPÇÃO BRASILEIRA Prof. Dr. Washington Lopes FIPECAFI / Trevisan AUDITORIA CONTÍNUA E SPED – SISTEMA PÚBLICO DE ESCRITURAÇÃO DIGITAL Cristiano Borges PwC "XBRL: O caso do Projeto SICONFI do Tesouro Nacional" Bruno de Sousa Simões Finance Analyst & Control Secretariat of the National Treasury

18h45 – 18h50

ENCERRAMENTO Coordenadores do Evento

18h50 – 22h30

JANTAR DE ENCERRAMENTO Golf Club São Francisco Bus departure from USP 19h00 return at 22h30 | Saída de ônibus da USP as 19h00 e retorno às 22h30

F) SESSÕES PARALELAS DE APRESENTAÇÃO DE TRABALHOS Para embasar o relatório relativo aos resultados das sessões de apresentação de trabalhos, a coordenação do CONTECSI solicitou que todos os moderadores de sessões preenchessem um formulário referente ao andamento da sessão comentando sobre a atualidade dos temas, a interação entre os presentes, os questionamentos levantados, comentários adicionais e demais questões relevantes sobre o desenvolvimento e resultado de cada sessão. A comissão organizadora do 11º CONTECSI agrupou os trabalhos aprovados pela comissão avaliadora do congresso em sessões de acordo com a área temática. Em alguns casos, houve necessidade de subdividir uma mesma área temática em duas ou mais sessões em razão da grande quantidade de trabalhos relacionados a um mesmo tema ou assunto. Por suas valiosas contribuições para o andamento das sessões paralelas de apresentação dos trabalhos cujos registros e comentários foram fundamentais para a elaboração deste relatório, a coordenação do CONTECSI agradece a todos os moderadores das sessões a seguir elencados: Eva Martha Chaparro Salinas, Maximiliano Osório de Vargas, Sérgio Antônio Pulzi Júnior, Mehran Misaghi, Wellington Sousa Aguiar, Adriana Vieira da Silva

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

483

Pissinato, Adalberto Fischmann, Eduardo Vicente de Sá, Rafael Timóteo de Sousa Júnior, Ygor Freitas de Almeida, Edson Sampaio de Lima, Maximiliano Vargas, Raquel Muniz, Adicinéia Aparecida de Oliveira, Emerson Antonio Maccari, Antônio José Balloni, Adriane Maria Arantes de Carvalho, Alexandre Borba Salvador, Maurício Leite, Alanley Severo Leite da Silva, Alexandre Morais de Souza, Edison Luiz Gonçalves Fontes, Ernani Marques dos Santos, Edilaine Soares, Ademir Macedo Nascimento, Michel Lens Seller, Adriano Dinomar Barp, Julio Alvarez Botello, Francisco Gilvan Lima Moreira, Ana Maria Nélo, Minerva Martínez Ávila, Liliana Giusti Serra, Alberto de Medeiros Júnior, Vilma Geni Slomski, Maria Irene da Fonseca e Sá, Marcos Ricardo Rosa Georges, Claudia Maria da Costa Archer, Renato Ribeiro Nogueira Ferraz, Gilberto Perez, João Carlos Hipólito, Rodrigo Barraco Marassi, Almir Carlos da Silva, Rafael Herden Campos, Marco Alexandre Terlizzi, Alessandro Marco Rosini, Jairo Simião Dornelas, Itiel Moraes da Silva, Érica Souza Siqueira, Denis Bruno Viríssimo, George Leal Jamil, Cesar Biancolino, Alessandro Marco Rosini, Nelma Zubek Valente, João Álvaro de Carvalho, Armando Malheiro e Denis Alves e Lima. SESSÕES PARALELAS Composição das Sessões – 11th CONTECSI - 2014

· · · · · · ·

Total 33 Parallel Sessions 14 Research Forum 3 Communication’s Forum Doctoral Consortium Post-Doctoral Consortium Master Colloquium 2 Poster Session

263 trabalhos 140 papers 60 papers 16 papers 8 projetos 2 projetos 7 projetos 15 posters

Apresentam-se a seguir, ordenados por data de ocorrência, os principais destaques e comentários a respeito das sessões paralelas de apresentação de trabalhos, registrados pelos moderadores das sessões, onde ocorreram as apresentações dos referidos trabalhos: May 28 | 28 DE Maio (WEDNESDAY | QUARTA-FEIRA) Na sessão 1(A), cujo tema central foi AIS – Accounting and Enterprise Information Systems, e moderação de Eva Martha Chaparro Salinas A. Quintans, foram apresentados os trabalhos: QUALITY OF ACCOUNTING SERVICES: IMPLEMENTATION OF STATEMENT ACCOUNTING FOR SMAILL AND MEDIUM ENTERPRISES de Silvana Dalmutt Kruger, Luciano Kruger e Sady Mazzioni; THE PUBLIC ACCOUNTANTS PERCEPTION IN THE REGION THAT ARE ASSOCIATED TO THE TOWN ASSOCIATION FROM THE MIDWEST FROM THE SANTA CATARINA STATE-AMMOC IN RELATION TO THE BRAZILIAN ACCOUNTING STANDARDS APPLIED TO THE PUBLIC SECTOR-NBCASP de Marcio Roberto

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

484

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

Piccoli e Roberto Carlos Klann; THE IMPORTANCE OF A ACCOUNTING INFORMATION SYSTEMS IN THE DECISION MAKING PROCESS OF BUSINESS de Igor Freitas de Almeida; RESEARCH SURVEY WITH THE ASSOCIATES OF THE SÃO PAULO CHAPTER OF PMI TO RAISE SITUATIONS AND INTERLOCUTORS OF THE MAIN PROBLEMS OF COMMUNICATION IN PROJECTS AND TO IDENTIFY CRITICAL SUCESS FACTORS de Airton Molena e Riccardo L. Roval. De acordo com a moderadora, os temas discutidos demonstraram ser de boa qualidade, pois foi apresentado o papel de um contador através dos sistemas de informações tanto em empresas privadas como públicas, além de abordarem as novas normas da cultura financeira e fiscal para que as empresas sigam adiante. Na sessão 1(B), que teve como tema INT – Internet and Virtual Communities and Social Networks, moderada por Maximiliano Osório de Vargas, apresentaram-se os seguintes trabalhos: THE EXPECTATION OF BRAZILIAN MANAGERS OF INFORMATION TECHNOLOGY ABOUT THE USE OF CLOUD COMPUTING de Alberto De Medeiros JR., Marcos Credidio Mello e Gilberto Perez; SENTIMENT ANALYSIS FOR IDENTIFICATION OF RISK IN THE ELECTRICITY SECTOR de Tiago Klein de Santana, Norton Trevisan Roman, Rafael Shoji e Juliana Makimi Kato; TRAFFIC PREDICTION, USING WT AND ARIMA, APPLIED TO ADAPTIVE MANAGEMENT BANDWIDTH FOR ROUTER INTERFACES de Newarney Torrezão da Costa, Jamil Salem Barbar, Fabíola Bento Soares, Miguel da Rocha Correia Lima e Colaboradora: Geycy Dyany Oliveira Lima; ASSESSMENT OF THE USABILITY OF A PUBLIC BANK'S SITE de Fabiano Siqueira Jesus, Eliete dos Reis Lehnhart, Vanderléia Leal Losekann e Mauri Leodir Lobler. Segundo o moderador da sessão, houve usabilidade prática dos temas apresentados, sendo esses de alta relevância e a temática discutida bastante atual. O trabalho PROPOSITION OF AN ALUMNI PORTAL BASED ON BENCHMARKING AND INNOVATIVE PROCESS de Gislaine Cristina dos Santos Teixeira e Emerson Antônio Maccari não foi apresentado devido ausência dos autores. A sessão 1(C), que teve como tema EDU- IS and IT Education and Curriculum Development, moderada por Sérgio Antônio Pulzi Júnior, apresentou os seguintes trabalhos: USABILITY ANALYSIS OF MOODLE – A COURSE MANAGEMENT TOOL (CMS) de Daniel da Cruz Santos e Reinaldo Lourenso; A MODEL FOR CROSS-CULTURAL EXPERIENTIAL LEARNING BASED ON BUSINESS GAME de Luiz Antônio Titton e José Dutra de Oliveira Neto; NEW TECHNOLOGIES IN HIGER EDUCATION: POSSIBILITIES AND LIMITS OF DISTANCE EDUCATION FROM THE POINT OF VIEW OF TEACHERS WHO WORK IN AN ACCOUNTING CLASSROOM COURSE de Vilma Geni Slomski, Alessandra Silva Santana Camargo, Adriana Maria Procópio de Araújo e Ronaldo Fróis de Carvalho; USE OF BLENDED LEARNING, CASE ESTUDY: AUTONOMOUS UNIVERSITY OF THE STATE OF MEXICO de María de La Luz Sánchez Paz, César Enrique Estrada Gutiérrez, Juan Alberto Ruíz Tapia, Julio Álvarez Botello e Colaborador: Minerva Martínez Álvila. Segundo o moderador, foram discutidos os tópicos: barreiras para mídias digitais no ensino; aspectos culturais, sociais e hábitos que dificultam o uso das mesmas e os paradigmas do EaD. A temática alcançou excelente nível de debate. A sessão 1(D), que teve como tema EGOV- E-Governance and Public Policies,

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

485

moderada por Mehran Misaghi, apresentou os seguintes trabalhos: DO THE MAYORS MEET THE LEGAL REQUIREMENTS OF ACCOUNTABILITY? A VIEW BASED ON THE ELECTRONIC PORTALS OF SANTA CATARINA’S MUNICIPALITIES de Fabiano Maury Raupp, Eliza de Abreu e Manuella Santos Silva; REFLEXOS DA IMPLANTAÇÃO DO SISTEMA DE TRATAMENTO DA NOTA FISCAL ELETRÔNICA SOBRE O CONTROLE DA COBRANÇA DO ICMS NA SECRETARIA DA FAZENDA DE PERNAMBUCO de Daniel Delgado Nepomuceno, Aldemar de Araújo Santos, João Gabriel Nascimento de Araújo e Tiago de Moura Soeiro, ANALYSIS OF THE EFFECTS OF STRATEGY USED FOR LOYALTY OF MEDICAL PROFESSIONALS IN PRIMARY CARE OF A BRAZILIAN CITY de Cristiano de Freitas Gomes e Chennyfer Dobbins Paes da Rosa, RESEARCH ON REDUCING COSTS OF TAX COMPLIANCE AND INVESTMENTS IN PUBLIC SYSTEM OF BOOKKEEPING DIGITAL – SPED IN BRAZIL de Edson Sampaio de Lima, Napoleão Verardi Galegale, Carlos Hideo Arima e Pedro Luiz Côrtes. Segundo o moderador, os temas abordados foram considerados atuais e, por esse motivo, geraram o interesse dos participantes e apresentadores. A sessão 1(E), que teve como tema ESD – Engineering and Software Development , moderada por Wellington Sousa Aguiar, apresentou os seguintes trabalhos: IT GOVERNANCE SOLUTION FOR THE INFORMATION TECHNOLOGY (IT) AREA OF BRAZILIAN PUBLIC FEDERAL SECTOR de Odnaldo Cruz Vieira Junior, Rejane M. da Costa Figueiredo e Elaine Venson; SOFTWARE METRICS: A SURVEY CONDUCTED WITHIN BRAZILIAN IT COMPANIES de Darlan F. de Arruda e José Gilson de A. T. Filho; ADDING QUALITY IN SOFTWARE WITH A PROPOSAL FOR INTEGRATION OF MODELS DRIVEN SOFTWARE DEVELOPMENT AND TESTING de Carla C. de Jesus Almeida e Adicinéia A. de Oliveira; AN APPLICATION FOR EDITING AND CREATION OF IMAGING AND STRIPS FOR THE SOCIAL NETWORKS USERS de Deivid do Vale Nascimento, Maiky Alves da Silva, Cristiane de Fátima dos Santos e Wesley Viana. Segundo o moderador, muitas pessoas estiveram presentes e as discussões atingiram alto nível de desempenho graças às diversas perguntas realizadas pelos ouvintes que puderam trocar informações sobre os diferentes comportamentos regionais. A sessão 1(F), que teve como tema ISM – Information Systems Management, moderada por Adriana Vieira da Silva Pissinato, apresentou os seguintes trabalhos:THE APPLICATION OF THE IEAC METHOD IN THE SUPPLIER MONITORING: A CASE STUDY IN THE NONWOVEN SECTOR de Douglas Schossler e Raquel Janissek-Muniz; CHALLENGES OF SMALL BUSINESSES RELATED TO LOGISTICS OPERATION OF ERP SYSTEMS de Renato B. de Lima Oliveira Júnior, Paulo Cristiano de Oliveira, José Abel de A. Baptista, Marcelo Rabelo Henrique e Colaborador: Marina Nakayama; BIG DATA USAGE IN THE MARKETING INFORMATION SYSTEM de Alexandre Borba Salvador e Ana Akemi Ikeda; INDICATORS MANAGEMENT: A CASE STUDY ON DIGITAL SERVICES COMPANY de Leonardo Fabris Lugoboni, Marcus Vinicius Moreira Zittei, Joice Chiareto e Tatiani Despirto Barroso; GOVERNACE CORPORATE TAX AFTER IMPLEMENTATION OF ELETRONIC GOVERNMENT de Marcus Vinicius Moreira Zittei e Antônio Fagner Gomes Amorim. Segundo a moderadora, as discussõs foram maduras e os questionamentos focaram na prática do BIGDATA. As principais questões

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

486

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

levantadas envolveram temas como: processos de governança e a relação com empresas de maneira geral, avaliação desempenho abordado como medida imprescindível para a evolução empresarial. A sessão 1(G) com a temática INV – IS and IT Innovation and Change, moderação de Adalberto Fischmann, teve a apresentação dos seguintes trabalhos: ADOPTION OF STORAGE DATA SYSTEMS IN CLOUD: A STUDY WITH END USERS de Adrianne Paula Vieira de Andrade, Gabriela Figueiredo Dias, Anatália Saraiva Martins Ramos e Manoel Veras de Souza Neto; BIG DATA: PUBLICATION EVOLUCION AND RESEARCH OPPORTUNITIES de Simone S. Luvizan, Fernando S. Meirelles e Eduardo H. Diniz; CONTEXT MAGEMENT BY EVALUATING QUALITY OF CONTEXT de Débora Cabral Nazario, Mario Antonio R. Dantas e José Leomar Todesco; E-RISK: A RISK MANAGEMENT MODEL FOR THE ESTRELA de Márcia Rodrigues S. Capellari, Lis Ângela de Bortoli, Amilton de Quadros Martins, Fahad Kalil, Colaboradores: Moacir da Silva Junior e Suellen Spinello Sotille. Segundo o moderador, os temas apresentados foram inovadores e trataram de transformações em sistemas de informação e na tecnologia em informação. Por isso, a platéia demonstrou elevado interesse em todos os trabalhos. A sessão 1(H) com o tema KGM – Knowledge Management And Business Intelligence, moderada por Eduardo Vicente de Sá, contou com a apresentação dos seguintes trabalhos: EXPANDING CONSUMER RESEARCH: SOCIAL MEDIA MONITORING STRATEGIES IN DISCOVERING OPPORTUNITIES FOR MARKETING PROJECTS de Rodrigues Sousa, Fernando José e Barbin Laurindo; REQUIREMENTS ASSESSMENT TO DATA WAREHOUSE USERS’ CLASSIFICATION de Stela de Queiroz, Pessoa Silvestre e Adriane Maria Arantes de Carvalho; THE PRACTICE OF KNOWLEDGE MANAGEMENT IN A SHARED SERVICES CENTER: A CASE STUDY ON A COMPANY IN THE BUSINESS OF RETAILING de Luciana Lehmann Holzhey e Raquel Janissek-Muniz. Segundo o moderador, as discussões tiveram um nível elevado. As principais questões levantadas foram sobre como gerenciar informações e, consequentemente, a gestão de conhecimento, bem como questões relacionadas ao apoio da alta gerência que é o primeiro passo para que as ferramentas e programas dêem certo. Os estudos mostraram a dificuldade de se armazenar e disseminar o conhecimento entre todos os níveis. A sessão 1(I) com o tema ITM – Information Technology Management, moderada por Rafael Timóteo de Souza Júnior, contou com a apresentação dos seguintes trabalhos: AN EVALUATION OF THE MANAGEMENT OF THE INFORMATION SYSTEMS AND TECHNOLOGIES IN HOSPITALS (GESITI-HOPITALS): NORTHEASTERN PLANNING REGION IN BULGARIA de Bistra Vassileva, Antonio José Balloni, Nadezhda Kalinova e Toni Avramov; SERVICE QUALITY MEASUREMENT: IMPLICATIONS FOR HEALTHCARE SECTOR IN BULGARIA de Bistra Vassileva e Antonio José Balloni; A TALE OF TWO STANDARDS: A PANAMERICAN STUDY OF THE FEASIBILITY OF AUDIT DATA STANDARDS FOR U.S.AND BRAZILIAN NOT-FOR-PROFIT ORGANIZATIONS de Stephen P. Kozlowski, Deniz Appelbaum, Denis Lima e Alves, Edson Luiz Riccio e Colaborador: Regis Michel dos Santos Sousa. Segundo o moderador, os trabalhos apresentados na sessão tratam de temas bastante atuais, o que possibilitou grande interesse dos ouvintes

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

487

e também uma rica discussão. A sessão 1(J) com o tema POSTER / PAINEL, moderada por Denis Alves e Alandey Severo Leite da Silva, contou com a apresentação dos seguintes trabalhos: DISCRETA 1,0: AMBIENTE COMPUTACIONAL DE APRENDIZAGEM DE MATEMÁTICA DISCRETA de Jefferson Biajone, Christian Raphael e Jonatas Vinicius Sales de Oliveira; A TEORIA DE GRAFOS NA MÚSICA CLÁSSICA E ERUDITA de Jefferson Biajone e Bruna Carolina de Almeida Gomes; DICA: DISPOSITIVO PARA INTERAÇÃO DE COMUNICAÇÃO AUMENTATIVA de Mário Ferreira da Costa, Ovídio José Francisco e Patrícia Nazaré Fonseca Motta; DISPLAY BRAILLE de Rodrigo Diniz, Ovídio José Francisco e Vitor Hugo Nogueira dos Santos; EXCOMBATENTES E TEORIA DE GRAFOS: OTIMIZAÇÃO DE CAMINHOS PARA VISITAÇÃO DE TÚMULOS DE HERÓIS NO CEMITÉRIO DE ITAPETININGA de Jefferson Biajone e Airesphy Pantojo Viotto; STUDY OF THE SHARES ON SOCIAL AND ENVIROMENTAL RESPONSIBILITY OF LISTED COMPANIES NINTH NEW MARKET de Graziela Oste Graziano, Valéria Rueda Elias Spers, Antonio Carlos Giuliani, Andrea Kassouf Pizziantto, Thel Augusto Monteiro e apresentado por Laudicério Aguiar Machado; A POPULARIZAÇÃO DA CIÊNCIA de Ana Nélo. Segundo os moderadores, as apresentações troxeram de forma prática e multidisciplinar as aplicações computacionais ao ensino, as necessidades especiais e a prática de funções organizacionais. A sessão contou com a presença de numeroso público e possibilitou alto nível de debate. A sessão 2(A) com o tema AIS – Accounting and Enterprise Information Systems, moderada por Ygor Freitas de Almeida, contou com a apresentação dos seguintes trabalhos: PROPOSAL O MANAGEMENT OF COSTS INCURRED IN A CALZONES INDUSTRY: A CASE STUDY de Fabiano Maury Raupp e Maria Eduarda Gonçalves Souza; SPED-PUBLIC SYSTEM OF DIGITAL BOOKKEEPING: ADAPTATIONS FOR USE AND IMPLEMENTATION BY ACCOUNTING PROFESSIONALS de Adriano Dinomar Barp, Adriana Torquato Vieira e Zilton Bartolomeu Martins; ANALYSIS OF THE RELATIONSHIP BETWEEN HEDGE ACCOUNTING PRACTICES AND AUDITING: A STUDY WITH BRAZILIAN COMPANIES AUDITED BY BIG FOUR FIRMS de Francisco Carlos Fernandes, Geovanne Dias de Moura e Rodrigo Barraco Marassi. Segundo o moderador, os trabalhos apresentados foram de um nível de discussão bem aprofundado, com temas atuais como: BIG FOUR, SPED e técnicas de análise de custos. Nas discussões, a principal contribuição deu-se em razão dos encaminhamentos individuais no sentido da necessidade de ampliar o campo de aplicação dos trabalhos apresentados. Não compareceu à sessão, o trabalho: COMPETITIVE INTELLIGENCE, CLOUD COMPUTING AND INNOVATION IN SMALL ENTERPRISES de Carlos Kono, Luiz Carlos Fraga e Silvia Jr e Leonel Cezar Rodrigues. A sessão 2(B) com o tema AUD – Systems Auditing and IT Governance, moderada por Edson Sampaio de Lima, contou com a apresentação dos seguintes trabalhos: WEAKNESSES ANALYSIS OF AUDIT REPORTS BASED ON PCAOB INSPECTION ON BUSINESS ACCOUNTING AND AUDIT AND BRAZILIAN AMERICAN 2010 TO 2012 de Franciele Wrubel, Tânia Cristina Chirello, Paulo Roberto da Cunha e apresentado por Márcio Roberto Piccoli; ANALYSIS OF

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

488

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

MATURITY LEVELS IN IT PROCESS RELATED TO INFORMATION SYSTEMS SECURITY de Elson Luciano Weber, Mariana Haefliger da Silva, Adolfo Alberto Vanti e Maria Cecília da Silva Brum; SYSTEMS AUDITING FOR ALTERNATIVE WORKING TIME SYSTEMS: BEST PRACTICES, VULNERABILITIES AND REQUIREMENTS FOR COMPLIANCE WITH BRAZILIAN REGULATIONS de Alessandro Santiago dos Santos, Leandro Avanço, Maria Cristina Machado Domingues e Denis Bruno Veríssimo; USING THE MODEL COSO ERM VERIFIABILITY FOR THE EFFECTIVENESS OF INTERNAL CONTROL PROCEDURES IN A COMPANY AGRO INDUSTRY de Silvana Dalmutt Kruger e Fábio Carlos Hammes; IT GOVERNANCE IN HEALTH INSURANCE COMPANIES – STUDY OF CASES IN STATE OF CEARÁ de Wellington Sousa Aguiar; Antônio Augusto Gonçalves, Cláudio Pitassi e Valter de Assis Moreno Júnior. Segundo o moderador, as questões mais discutidas referiram-se ao processo de geração de valor percebido pelo uso de governança de TI nas empresas e a preocupação em utilizar governança corporativa e de TI, muito mais para atender atos regulatórios do que aos usuários da informação comuns. Na sessão 2(C), dedicada ao tema INV – IS and IT Innovation and Change, a moderação foi feita por Maximiliano Vargas e contou com os seguintes trabalhos: DIGITAL CITY PROJECTS: ANALYSIS OF THE CITIES OF ALMIRANTE TAMANDARÉ, CURITIBA, FAZENDA RIO GRANDE, PINHAIS AND SÃO JOSÉ DOS PINHAIS de Maximiliano Osório de Vargas e Denis Alcides Rezende; RISK REGISTER ON IT PROJECTS IN THE BANKING SECTOR: AN EXPLORATORY STUDY de Marco Alexandre Terluzzi, César Augusto Biancolino, Cláudia Terezinha Kniess e Evandro Prieto; MAPPING OF BRAZILIAN SCIENTIFIC PRODUCTION INVOLVING KNOWLEDGE MANAGEMENT IN THE WEBQUALIS SYSTEM FROM 1999 TO 2012 de Maria Zenilda da Silva, Flávia de Martins Faria Vieira Domingues, Arturo Muttoni Deambrosis, Clarissa Carneiro Mussi e Colaboradora: Cristina Martins; COMPETITIVE INTELLIGENCE AT DIGITAL MARKETING de Fernando José Laurindo Barbin e Lidia Gimenez Simão Macul. Segundo o moderador, o nível de discussões foi muito bom, pois as temáticas abordadas são modernas e com aplicabilidade garantida na atualidade. Na sessão 2 (D) cujo tema foi ISM - Information Systems Management, a moderação ficou a cargo de Raquel Muniz para coordenar as apresentações dos seguintes trabalhos: ANALYZING THE IMPACT OF CUSTOMIZATIONS IN WMS MODULE ON THE IMPLEMENTATION OF AN ENTERPPRISE RESOURCE PLANNING – ERP de André Gorbato, Paulo Cristiano de Oliveira, José Abel de Andrade Baptista, Marcelo Rabelo Henrique e Colaboradora: Marina Keiko Nakayama; IPV6: AN ANALYSIS OF IMPACTS DUE TO THE CHANGE OF IP ADDRESSING PROTOCOLS IN ORGANIZATIONS de George Leal Jamil, Claudio Roberto Magalhães Pessoa, Thiago Almeida Santos Geremias e Bruno Mattos; MARKETING INFORMATION SYSTEM APPLIED TO RETAIL (MISR): A THEORICAL PROPOSAL de Katherine B.G.B. Sresnewsky, Ana A. Ikeda, Andres Veloso e Guilherme F. Shiraishi; EVALUATION OF INFORMATION SYSTEMS FOCUSING ON USERS’ SATISFACTION: A STUDY DEVELOPED AT METROPOLITAN REGION OF BELO HORIZONTE – MG de Antônio Artur de Souza, Lorena Gouvea Alves, Amanda Lucchesi Lara, Dirceu da Silva e Colaboradores: Ewerton Alex Avelar e André Sousa Braga; ANALYSIS OF

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

489

INFORMATION FLOWS OF THE COST MANAGEMENT SYSTEMS IN HOSPITALS de Antônio Artur de Souza, Ciro Gustavo Bragança, André Sousa Braga e Ewerton Alex Avelar. Segundo o moderador, as discussões foram muito importantes para realçar os principais aspectos dos trabalhos apresentados. As questões foram pertinentemente relacionadas à gestão da informação com participação ativa da platéia. Na sessão 2(E) cujo tema foi ITM – Information Technology Management, a moderação ficou a cargo da Adicinéia A. de Oliveira para coordenar as apresentações dos seguintes trabalhos: IT SERVICE MANAGEMENT: A CASE STUDY IN CASA DAS BALAS LTDA de Lucas Sampaio Leite, Glaydson Martins do Vale e Assuero Fonseca Ximenes; SUPPLY INPUTS IN INTENSIVE CARE UNITS (ICU) AFFECTED BY INFECTIONS RELATED TO HEALTH (IRAS): WHAT IS THE RELEVANCE TO MANAGERS OF HEALTH? de Walquiria Barcelos de Figueiredo; ROLE OF MANAGEMENT IN IDENTIFICATION OF RISK FACTORS SAFETY OF PATIENTS IN RESPIRATORY SUPPORT INVASIVE de Edzângela de Vasconcelos Santos e Márcia Cristina Zago Novaretti; E-COMMERCE AS A BUSINESS STRATEGY TO VALLEY OF TOLUCA BUSINESSES, MEXICO de Julio Álvarez Botello, Eva Martha Chaparro Salinas, María Del Carmen Hernández Silva e Carlos René García Vivas. De acordo com o moderador, todos os participantes presentes contribuíram para o sucesso da sessão com questões e apontamentos relevantes aos temas abordados. O trabalho STUDY OF THE USE OF TECHNIQUE ROI PROJECT INFORMATION TECHNOLOGY FOCUS ON THE FOUR MAJOR SUB-AREAS de Hugo Leonardo Miranda, Marco Antônio de Sousa, Tomás Antônio Costa Badan e Adriano C. Santana não foi apresentado devido à ausência dos autores. Na sessão 2(F), cujo tema foi HEA – Healthcare Information Systems and Technology, a moderação ficou a cargo de Emerson Antônio Maccari para coordenar as apresentações dos seguintes trabalhos: THE VENTILATOR-ASSOCIATED PNEUMONIA AS A TRACER CONDITION FOR QUALITY AND PATIENT’S SAFETY IN THE ADULT INTENSIVE CARE UNIT. A CRITICAL PROCESS ANALYSIS de Sérgio Antônio Pulzi Júnior, Vanessa da Silva Teixeira, Claudia Regina Cachulo Lopes e Milton Soibelmann Lapchik; THE IMPLEMENTATION OF NURSING PROCESS ALLOWS IMPROVING THE QUALITY OF RECORDS AND THE CARE GIVEN de Flavia Regina Cocuzza das Eiras e Antonio Pires Barbosa; IMPACT OF AMBULATORY MEDICAL CARE UNITS (AMA) ABOUT THE DEMAND ON THE NETWORK OF HEALTH CARE: A QUANTITATIVE STUDY ON ITS TRAJECTORY FROM 2005 TO 2012 de Adriana Vieira da Silva Pissinato e Lara Jansiski Motta. Segundo o moderador, a sessão ocorreu normalmente, considerando-se a pontualidade e discussões sobre os temas. Na sessão 2(G), cujo tema foi ITM – Information Technology Management, a moderação ficou a cargo de Antônio Jose Balloni para coordenar as apresentações dos seguintes trabalho: ANALYSIS OF CONTRACTS OF IT OUTSOURCING SERVICES IN TCU AND LEGISLATION GOVERNING THESE CONTRACTS IN FEDERAL PUBLIC ADMINISTRATION de Graziela Ferreira Guarda , Edgard Costa Oliveira e Rafael Timóteo de Sousa Junior; IT PROJECT MANAGEMENT SUCCESS: THE INFLUENCE OF PROJECT MANAGER COMPETENCIES AND TEAM COMMITMENT de Cíntia Cristina Silva De Araújo e Cristiane Drebes Pedron;

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

490

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

CURRENT STAGE OF CLOUD COMPUTING ADOPTION ON HIGHER EDUCATION INSTITUTIONS IN THE STATE OF RIO GRANDE DO NORTE de Thiago Fernandes Silva Dutra, Adriane Araújo de Oliveira e Manoel Veras de Sousa Neto. Segundo o moderador, os temas abordados despertaram interesse nos ouvintes que participaram de forma bastante interativa com questões que se referiam a: falta de planejamento de longo prazo em relação a serviços de TI, bem como a gestão no âmbito governamental; analisar a visibilidade da substituição da prestação de serviços terceirizados de TI por servidores de carreira, fortalecendo a gestão e qualificação dos profissionais e dos órgãos que os utilizam. Na sessão 2(H) cujo tema foi PRM – Project Management, a moderação ficou a cargo de Adriane Maria Arantes de Carvalho para coordenar as apresentações dos seguintes trabalhos: INFORMATION TECHNOLOGY (IT) PROJECT USING VIRTUAL TEAMS: CRITICAL SUCCESS FACTORS AND THEIR LEVEL OF IMPORTANCE de André Machado Dias Ferreira e Fernando José Barbin Laurindo; RISK-TAKING AND RISK MANAGEMENT OF PROJECTS: A STUDY IN A CONSULTING FIRM OF IT de Ana Claudia Belfort e Cristina Dai Prá Martens; BALANCING STRATEGY AND TACTICS IN PUBLIC PROJECT IMPLEMENTATION: A CONTINGENCY APPROACH de Leandro Ranolfi Girardi e Roque Rabechini Junior; A DESCRIPTIVE STUDY ON THE CRITICAL SUCCESS FACTORS IN THE IMPLEMENTATION OF A PROJECT OFFICE IN AN INSTITUTION OF HIGHER EDUCATION de Eduardo Vicente de Sá e Leandro Alves Patah; IMPACT OF INDICATORS GDP AND UNEMPLOYMENT RATE ON DEMAND PASSENGER OF LINE 3 ON THE SÃO PAULO SUBWAY de Vagner Sanches Vasconcelos e Daniel Reed Bergmann. De acordo com o moderador, a platéia foi bastante participativa em todas as apresentações. Os trabalhos apresentados enfocaram aspectos distintos da gestão de projetos, tais como os fatores de sucesso, balanceamento estratégico e tático, além de estabelecerem uma ponte com outras temáticas como a assunção de riscos e a definição de indicadores. Destacou-se a percepção da importância da visão estratégica da gestão de projetos no setor público e também privado. Percebeu-se a emergência de novos temas e propostas de pesquisa evidenciando a atualidade das temáticas debatidas. Na sessão 2(J) cujo tema foi POSTER, a moderação ficou a cargo de Denis Lima e Alves para coordenar as apresentações dos seguintes trabalhos: AUTOMATIZAÇÃO DE LIXEIRAS COM COLETA SELETIVA de Marcus Vinicius Branco de Souza e Leonardo Felipe Faria do Amaral; PROJETO GIAC: GERENCIADOR INTEGRADO DE ABASTECIMENTO E CONTROLE de Marcus Vinícius Branco de Souza e Ezaias Rodrigues dos Santos; PETS APP: APLICATIVO MOBILE PARA ADOÇÃO E CRIAÇÃO DE ANIMAIS DE ESTIMAÇÃO de Andréia Rodrigues Casare, Larissa Oliveira Simões, Wayne Almeida Da Conceição, Samara Rached Souza e Felipe Augusto Modesto; APLICATIVO CLOUD SHOP. INTEGRANDO O CLIENTE AOS SERVIÇOS DO SHOPPING: ALIMENTAÇÃO, LOCALIZAÇÃO E COMPRAS. TUDO EM SUA MÃO! de Marcus Vinícius Branco de Souza, Danilo Renan Machado Queiroz, Eduardo Hideki Nagata e Lucas Souza de Oliveira; COTAGRO: APLICATIVO PARA CONSULTA DE COTAÇÕES AGROPECUÁRIAS de Danilo Ruy Gomes e Rodrigo Alves da Silva; FIELDPLAN: APLICATIVO PARA GERENCIAMENTO DE CENTRO ESPORTIVO de Danilo Ruy Gomes, Fabiana Paula De Goes e Patrícia Nazaré Fonseca Motta; QRCOW: APLICATIVO PARA

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

491

ACESSO A INFORMAÇÕES SOBRE ANIMAIS EM EXPOSIÇÕES E LEILÕES de Marcus Vinícius Branco de Souza, Ivan Rodrigues de Camargo, Leandro Kimura Rodrigues de Moraes, Lidia Eliza Siqueira Panis e Luiz Henrique Machado Mello; SPARA – SUA VIDA NÃO ESPERA! SISTEMA BASEADO EM INTELIGÊNCIA ARTIFICIAL PARA ENCAMINHAMENTO DE RESGATE de Marcelo dos Santos Silvério, Danilo Renan Machado Queiroz, Giuliano Augusto Zacarias e Carlos Luiz Ferreira Neto; ANJO APP: APLICATIVO PARA SUPERVISÃO DE CRIANÇAS E ANIMAIS DE ESTIMAÇÃO de Marcus Vinicius Branco de Souza, Jefferson Nunes de Paiva, Fabiana Paula de Goes e Patrícia Nazaré Fonseca Motta. Segundo o moderador, os trabalhos apresentaram propostas inovadoras, com muitas possibilidades de aplicação prática. Questionou-se a necessidade de registro de patentes, não mencionada nos trabalhos e também sobre a viabilidade de mercado dos projetos, que, na maioria, demonstraram maturidade analítica. Na sessão 2(J) cujo tema foi COMM – Research Communication, a moderação ficou a cargo de Alexandre Borba Salvador para coordenar as apresentações dos seguintes trabalhos: IT SERVICES MANAGEMENT IMPLEMENTATION WITH ADOPTION OF THE GOOD PRACTICES OF ITIL® V3 de José Luiz da Silva, Bruna Paiva da Silva e Fernanda Helena Rosa Pereira; SUSTAINABLE MANAGEMENT: A STUDY ABOUT SOCIAL AND ENVIRONMENTAL SUSTAINABILITY LEVEL OF A FEDERAL INSTITUTION OF HIGHER EDUCATION de Ines Liani Menzel Warken, Veridiana Jéssica Henn e Fabriacia Silva da Rosa; INTRODUCTORY ANALYSIS OF THE RESEARCH ABOUT THE COLLABORATIVE VIRTUAL ENVIRONMENT OLHAR EDUCADOR de Pedro Volcov e Ligia Capobianco; PROPOSAL OF AN INTEGRATED TOOL TO FACEBOOK FOR ASSIST IN PREPARATION THE NATIONAL EXAMINATION FOR STUDENTS – ENADE de Paulo Rogério Pires Manseira e Mehran Misaghi; MOBILE TECHNOLOGY USE OVERLOAD: PROPOSED ELEMENTS FOR A STUDY WITH FINANCIAL MARKET PROFESSIONALS de Carla Bonato Marcolin, Henrique Freitas e Ariel Behr. Segundo o moderador, as discussões foram ricas.

MAY 29 / 29 DE MAIO (THURSDAY / QUINTA-FEIRA) A sessão 3(A), com o tema AIS – Accounting and Enterprise Information Systems, teve como moderador Maurício Leite e contou com a apresentação dos trabalhos: IMPLEMENTATION PUBLIC DIGITAL BOOKKEEPING SYSTEM SPED: CHANGES IN EVERYDAY PERCEPTION OF ACCOUNTING PROFESSIONALS de Adriano Dinomar Barp, Maria Mônica da Silva Fortes e Daniel Sonaglio; RELATIONSHIP BETWEEN THE CHARACTERISTICS OF THE AUDIT COMMITTEE TO THE LEVEL OF DISCLOSURE OF INTELLECTUAL CAPITAL de Paulo Roberto da Cunha e Vanderlei Gollo; FINANCIAL INVESTMENT: A REVIEW OF INVESTOR PROFILE OF THE UNIVERSITY OF BAHIA RECÔNCAVO de Adeíse Caldas Dos Santos, Edna Araujo de Souza e Ivo Pedro Gonzalez Junior; PUBLIC-PRIVATE EQUITY: A CASE ANALYSIS OF THE APPLICABILITY OF TOLLS IN RIO GRANDE DO SUL de Nadia Mar Bogoni, Sílvia Letícia Bampi, Alícia Cechin e Amanda Guareschi; SUPPLY CHAIN MANAGEMENT: BARRIERS TO IMPLEMENTATION IN A CLUSTER OF

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

492

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

SMALL CITY de Telma Aline Torricelli, Hamilton Pozo e Takeshy Tachizawa. Segundo o moderador, a sessão ocorreu dentro da normalidade, sendo possível bom nível de discussão com temáticas atuais e trabalhos pontuais. A sessão 3(B), com o tema XBRL- XBRL and Enterprise Onthology, teve como moderador Cecília Moraes Santostaso Geron e contou com a apresentação dos trabalho: THE EFFECT OF THE USE OF XBRL BY THE BRAZILIAN COMPANIES IN DECREASING ASYMMETRY OF INFORMATION de Cecília Moraes Santostaso Geron, Roberta Auler Bittencourt, Edson Luiz Riccio, Marici Cristine Gramacho Sakata e Colaboradora: Marta Cristina Pelucio Grecco; A DATA MODEL FOR SPED BASED ON XBRL GL de José Geraldo Luciano e Paulo Caetano da Silva; THE ORGANIZING VISION OF XBRL IN BRAZIL de Maurício Taufic Guaiana e Edson Luiz Riccio; XBRL MOBILE: A TOOL FOR VIEWING AND MANIPULATION OF DATA ON MOBILE DEVICES XBRL de Ubirajara da Costa Lima Júnior e Paulo Caetano da Silva; RISKS AND BENEFITS OF USING XBRL: A STUDY BIBLIOGRAPHIC de Patrícia Kozoroski Possani, Maria Cecília da Silva Brum, Elson Luciano Weber e Débora Skibisnki Assumpção. Segundo o moderador, os participantes tiveram condições de discutir problemas relevantes, referentes ao tema XBRL através da elaboração de questões como: o efeito do uso de XBRL com a taxonomia IFRS: os incentivos das empresas e países para adotar a tecnologia: a falta de padronização, já que as taxonomias podem ser estendidas pelos usuários: a falta de conhecimento no Brasil, bem como a apresentação de um aplicativo para celular para visualizar e tratar arquivos XBRL. Na sessão 3(C), cujo tema foi ICT – ICT4DEV – Information and Communication Technologies for Development and Assistive Technologies, o moderador foi Alexandre Moraes de Souza. Foram apresentados os seguintes trabalhos: THE IMPORTANCE OF DATABASE MARKETING (DBM) FOR THE CUSTOMER MANAGEMENT de Débora Pires Otoni, Márcio Aurélio Ribeiro Moreira e Adriana Medeiros Martins; RISK MANAGEMENT IN AGILE METHODOLOGIES de Allan Kardec Silva Soares e Márcio Aurélio Ribeiro Moreira; METADATA FORMATS FOR ELECTRONIC PUBLICATION DESCRIPTION de Liliana Giusti Serra e José Fernando Modesto da Silva; STUDY OF FACETED ANALYSIS THEORY AND CONCEPT MAPS FOR ORGANIZATION OF LEARNING OBJECTS REPOSITORY de Cristiane Mendes Netto e Gercina Ângela Borém Oliveira Lima; FACTORS DETERMINING THE ACQUISITION OF iPAD BY CONSUMERS de Gilberto Perez, Juliana Ferreira Maldonado, Alberto Medeiros Jr. e Victor Freitas de Azerêdo Barros. Segundo o moderador, todos os artigos trouxeram temas de revelância para a área de gestão de TI, especialmente estudos voltados para a gestão de riscos em metodologias ágeis, DBM, e análise facetada que, de fato, agregam com a experiência aplicada em uma visão para desenvolvimento nos negócios de TI e para melhoria no gerenciamento da informação. A sessão 3(D), moderada por Edison Luiz Gonçalves Fontes, com o tema EDU-IS and IT Education and Curriculum Development contou com a apresentação dos trabalhos: CHARACTEREOLOGICAL USE OF ROGER VERDIER FOR CREATING LEARNING GROUPS IN SOCIAL NETWORKS WITH SOCIAL AFFINITIES de Antônio Carlos Pantaleão Rabelo Júnior, José Ahirton Batista Lopes Filho, Ivone Ascar Sauáia Guimarães e Will Ribamar Mendes Almeida; INSTRUCTIONAL DESIGN FOR

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

493

THE INCORPORATION OF LEARNING UNIT "DIFFERENTIAL AND INTEGRAL CALCULUS" IN LOGISTICS DEGREE IN DISTANCE EDUCATION de Minerva Martínez Ávila, Julio Álvarez Botello e Eva Martha Chaparro Salinas; THE IMPACT OF TECHNOLOGY IN TEACHING METHODS: A CASE STUDY COURSE LATO SENSU MUNICIPAL PUBLIC MANAGEMENT de Ana Maria Nélo, Antonio José Silva Oliveira e Francisco Gilvan Lima Moreira. Segundo o moderador, o nível de discussão foi amplo e bastante proveitoso, com ótima participação dos ouvintes. O trabalho ORGANIZATIONAL COMPETENCES TO RECOGNIZE DISTANCE LEARNING COURSE IN HIGHER EDUCATION: A STUDY ON THE UNIFIED INSTRUMENT OF EVALUATING MINISTRY OF EDUCATION – MEC de Aurélio Fiorillo, Emerson Antonio Maccari e Cibele Barsalini Martins, não foi apresentado devido à ausência dos autores. A sessão 3(E), com o tema EGOV-E–Governace and Public Policies, foi moderada por Ernani Marques dos Santos e contou com a apresentação dos trabalhos: AN ANALYSIS OF CITIZENSHIP PROGRAM FISCAL STIMULUS IN BRAZILIAN STATES AND FEDERAL DISTRICT IN THE PERIOD 2004 TO 2012 de Nicole Stefani Campos Ramos, Sérgio Murilo Petri, Luiz Felipe Ferreira e Priscila Alano da Rosa; OPERATIONAL CHANGES IN IMPLEMENTATION OF SPED BOOK STUDY IN A COMPANY PRODUCTS FOR DIAGNOSTICS de Marcos Felipe Kretzer de Souza, Sérgio Murilo Petri, Maria Denize Henrique Casagrande e Jacir Leonir Casagrande; FRAMEWORK FOR DEVELOPMENT OF CMS FOR SYSTEMIC DIRECTORS de Jairo Marotta Neto, Rafael Arlindo Dias, Gustavo Henrique Da Rocha Reis e Frederico de Miranda Coelho; IMPACTS OF ENTERPRISE RESOURCE PLANNING (ERP) FROM THE PERSPECTIVE OF COSO IN A LARGE COMPANY OF SOUTHERN BRAZIL de Francisco Carlos Fernandes e Lara Fabiana Dallabona; PROMOTING FISCAL CITIZENSHIP THROUGH E-TAXATION de Rute Abreu, Fátima David, Liliane Cristina Segura e Henrique Formigoni. Segundo o moderador, todos os trabalhos apresentados foram bastante comentados e questionados pela platéia. As temáticas foram muito atuais e as discussões giraram em torno da resistência a mudanças e a influência das culturas locais e organizacionais nos processos de implementação das tecnologias e sistemas. Na sessão 3(F), com o tema ESD – Engineering and Software Development, que teve como moderador Edilaine Soares e contou com a apresentação dos trabalhos: USE OF PLATFORMS IN PRODUCT DEVELOPMENT: A CASE STUDY IN BRAZILIAN SOFTWARE INDUSTRY de Ana Lúcia Figueiredo Facin, Mario Sergio Salerno e Mauro de Mesquita Spinola; FREE SOFTWARE EFFECTIVE USE: A SOFTWARE LIBRARY STUDY CASE de Rafael Arlindo Dias, Mychael Myk Assis Macedo e Frederico De Miranda Coelho; AGILE SELF-MANAGEMENT TEAMS: A SYSTEMATIC MAPPING ABOUT SOFT SKILLS de Keli Cristina V. S. Borges, Laís Barreto de Araújo, Aline Lopes Timóteo e Rejane M. da Costa Figueiredo; RISK MANAGEMENT IN AGILE SYSTEMS DEVELOPMENT ALIGNED WITH THE BEST PRACTICES OF IT GOVERNANCE de Manuel Ferreira da Costa Filho, Wellington Sousa Aguiar, Otávio Fernandes Frota, Leandro de Araújo Herculano e Colaborador: José Mário Bezerril Fontenelle; A SYSTEMATIC REVIEW OF THE MAIN PRODUCT OWNER’S FEATURES de Vinicius Maia de Brito, George Marsicano Corrêa e Wander Cleber Maria Pereira da Silva. O moderador relatou que as

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

494

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

discussões ocorreram em nível execelente, abordando aspectos teóricos e metodológicos. As discussões abordaram também os aspectos da arquitetura da plataforma integradora e os métodos ágeis e software. A sessão 3(G), com o tema INV – IS and IT Innovation and Change, que teve como moderador Ademir Macedo Nascimento, contou com a apresentação dos trabalhos: THE E-PARTICIPATION AS A TOOL FOR SOCIAL CHANGE - A CASE STUDY de Anselmo Luiz Éden Battisti, Maria Alexandra Cunha e Marcio Rodrigues Da Fonseca; THE RESOURCE-BASED VIEW IN DEVELOPMENT OF COMPETITIVE ADVANTAGES OF ORGANIZATION OF INFORMATION TECHNOLOGY AND COMMUNICATION OF PARTICIPANT LOCAL PRODUCTIVE ARRANGEMENT de Marcio Rodrigues da Fonseca, Fernando Moura Palhares e Maria Alexandra Cunha; APPROACH REGARDING THE CONTROLLER AND MANAGERIAL ACCOUNTING: A STUDY OF SOCIAL NETWORKS IN INTERNATIONAL JOURNAL PUBLISHED de Carlos Eduardo Facin Lavarda, Maria Margarete Baccin Brizolla, Tânia Cristina Chiarello e apresentado por Rodrigo B. Marassi; EVOLUTION OF CERTICS REFERENCE MODEL FOR SOFTWARE RESULTING FROM TECHNOLOGICAL DEVELOPMENT AND INNOVATION IN THE COUNTRY de Clenio F. Salviano, Angela M. Alves, Giancarlo N. Stefanuto e Sônia T. Maintinguer; APPLICATION OF GEOGRAPHIC INFORMATION SYSTEMS IN THE AGRICULTURAL SECTOR OF THE STATE OF MEXICO. CASE TROUT PRODUCTION IN THE STATE OF MEXICO de Elsa Mireya Rosales Estrada e Francisco José Holguín García. Segundo o moderador, a sessão ocorreu dentro da normalidade, com tempo para apresentação e discussão de todos os trabalhos. Na sessão 3(H), cujo tema foi PRM – Project Management, a moderação foi realizada por Michel Lens Seller, e ocorreram as apresentações dos seguintes trabalhos: INFORMATION SECURITY MANAGEMENT: A CASE STUDY IN A BRAZILIAN FINANCIAL INSTITUTION de Francisco Carlos Fernandes, Antonio Maria Da Silva Carpes e Elisandra Henn Diel; RELATIONSHIP BETWEEN CONTEXTUAL VARIABLES OF DEVELOPMENT PRODUCT PROJECTS, THE LEADERSHIP PROFILE, AND THE RESULT OF THE PROJECT de Dinah Eluze Sales Leite e Claudiano Sales Araujo; THE PROJECT SELECTION IN A COMPANY PROVIDING ENGINEERING SERVICES de Nestor De Oliveira Filho e Franciane Freitas Silveira; PROJECT MANAGEMENT MATURITY AND THE INTERNATIONALIZATION PROCESS: A CASE STUDY IN AN INTERNATIONAL COMPANY de André Luiz Spinelli Schelini, Cristina Dai Prá Martens e Marcos Roberto Piscopo; PROJECT PORTFOLIO MANAGEMENT MATURITY AND THE INFORMATION TECHNOLOGY ROLE IN ORGANIZATIONS: AN EMPIRICAL STUDY de Heverton Roberto Oliveira Cesar de Moraes e Marcos Paixão Garcez. Segundo o moderador, todos os trabalhos geraram boas discussões. A sessão 4(A) com o tema AIS – Accounting and Enterprise Information Systems, teve como moderador Adriano Dinomar Barp, e a apresentação dos artigos: AN ACCOUNTING ERROR OR FRAUD - WHAT IS THE CONTEXT? de Joshua Onome Imoniana, Carine Valverde Pietri, Mariana Pimentel do Rosário e Thaís Pedrosa; DISCUSSION ABOUT CARBON CREDITS IN THE BRAZILIAN FEDERAL AND MUNICIPAL TAX CONTEXTS de Silvana Karina De Melo

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

495

Travassos, Jose Isidio De Freitas Costa e Luis Carlos Miranda; ANALYSIS OF FINANCIAL AND ECONOMIC PERFORMANCE TECHNOLOGY COMPANIES LEVERAGING INFORMATION BRAZILIAN AND CHILEAN de Jaqueline Carla Guse, Maurício Leite, Tarcísio Pedro da Silva e Vania Regina Morás; DETERMINANTS OF THE CAPITAL STRUCTURE COMPANIES IN INFORMATION TECHNOLOGY BRAZILIAN de Itzhak David Simão Kaveski, Larissa Degenhart, Mara Vogt e Tarcísio Pedro da Silva. Segundo o moderador, o nível das discussões foi elevado, os temas muito atuais e relevantes para as temáticas abordadas. Houve ativa participação da plateia que elaborou questões fundamentadas como: corrupção e fraude em alto escalão têm o mesmo sentido?; um comitê de auditoria ajudaria ou não a minimizar a fraude?; diferenças entre corrupção ativa e passiva; porque no Brasil não funciona na questão dos créditos de carbono?; por que esses créditos não são tributados? Na sessão 4(B), com o tema SOC – Social Issues in IS and IT, o moderador foi Júlio Alvarez Botello para a apresentação dos trabalhos: RESISTANCE DETERMINED BY INTERACTION: A ROBUST THEORETICAL EXPLANATION de Jorge Lheureux de Freitas, Cristiano Ramos Moreira e Mauricio Gregianin Testa; PARADOXES THE USE OF MOBILE TECHNOLOGY: A STUDY WITH PROFESSIONAL USERS OF SMARTPHONE de Aline Carrion Mendieta, Cristina Dai Prá Martens e Ana Claudia Belfort; FACEBOOK AND CORPORATE COMMUNICATION: A NETNOGRAPHIC ANALYSIS OF THE USE OF SOCIAL NETWORKING WITH THE CASE OF BANCO SANTANDER de Marconi Freitas-Da-Costa, Dalton Luiz Leça Silva e Jordana Soares de Lira; ANALYSIS OF SYNERGY OF TECHNOLOGY PARKS WITH THE INFLUX OF HUMAN DEVELOPMENT OF BRAZILIAN MUNICIPALITIES de Alandey Severo Leite Da Silva, Ed Porto Bezerra e Filipe Araujo; MOBILE MONEY, YOUR APPLICATION IN THE WORLD AND IN BRAZIL de Marcílio Da Silva Correia, Bruno Moritoshi Arashiro e Tania Pereira Christopoulos. Segundo o moderador, os temas abordados são muito atuais e as pesquisas realizadas apresentadas foram bastante avançadas e de excelente qualidade. A sessão ocorreu pontualmente, possibilitando apresentação e discussão de todos os trabalhos previstos no programa. Na sessão 4(C), com o tema SEC – IS and IT Security, a moderação foi realizada por Francisco Gilvan Lima Moreira, que coordenou a apresentação dos trabalhos: ALIGNMENT OF INFORMATION SECURITY WITH BUSINESS AREAS CONTRIBUTION OF NBR ISO/IEC 27002:2013 de Edison Luiz Goncalves Fontes; SCIENTIFIC PRODUCTION ABOUT INFORMATION SECURITY ON BRAZILIAN SCIENTIFIC CONFERENCES de Antonio Eduardo de Albuquerque Junior e Ernani Marques dos Santos; ANALYSIS OF VULNERABILITIES IN WIRELESS NETWORKS IN THE MALLS OF SÃO LUÍS – MA, USING THE TECHNIQUE WARDRIVING de Arlley Carlos Carneiro dos Santos Costa e Will Ribamar Mendes Almeida; GP COMPLETE: A MOBILE SOLUTION TO HELP MANAGE PROJECTS de Igor Fontes Novais, Monique Luiza S. do R. Dantas e Paulo Caetano da Silva. Segundo o moderador, a sessão ocorreu dentro da normalidade, sendo pontualmente efetivada como previsto no programa. A sessão 4(D), cujo tema foi INT – Internet and Virtual Communities and Social

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

496

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

Networks, teve como moderadora Ana Maria Nélo e contou com a apresentação dos trabalhos: MODERNIZATION OF LEGACY SYSTEMS TO SOA (SERVICE ORIENTED ARCHITECTURE): A CASE STUDY IN A COMPANY FROM THE SERVICES INDUSTRY de Michel Lens Seller e Colaborador: Fernando J. B. Laurindo; AROUSAL AND CONTENT UTILITY AS EMOTIONAL INFLUENCERS OF SHARING INFORMATION AMONG ONLINE CONSUMERS de Wilian Ramalho Feitosa e Delane Botelho; ADOPTION OF SOCIAL NETWORKS SITES: MENTAL CONSTRUCTS USED BY EXPERIENCED USERS OF ARACAJU de Ademir Macedo Nascimento, Debora Eleonora Pereira da Silva e Antonio Luiz Rocha Dacorso; COLLABORATIVE NETWORKED ORGANIZATIONS AND THEIR FORMS OF OPERATION - A STUDY APPLIED ON A SCIENTIFIC EDITOR de Alexandre Rocha De Azevedo e Marco Antonio Pinheiro Silveira; THE RELATION BETWEEN THE COMMENTS FROM USERS ON FACEBOOK AND FUTURE PURCHASE DECISION: AN EXPLORATORY STUDY de Evangelina de Melo Bastos, Anatália Saraiva Martins Ramos, Adrianne Paula Vieira de Andrade e Larissa Mayara da Silva Damasceno; SERVICES INNOVATION: CUSTOMER CONTRIBUTION IN VIRTUAL SOCIAL NETWORKS de Wagner Amaral e Silva e Débora Eleonora Pereira da Silva. Segundo a moderadora, todas as apresentações geraram dúvidas, contribuições e sugestões. Os debates fundamentaram-se nos seguintes temas: redes sociais com predomínio do Facebook, a metodologia aplicada e as limitações das pesquisas. Na sessão 4(E), com o tema EDU – IS and IT Education and Curriculum Development, moderada por Minerva Martínez Ávila, foram apresentados os trabalhos: PROPOSAL FOR IMPLEMENTATION OF TECHNOLOGICAL RESOURCES DIGITAL TOUCHSCREEN IN EDUCATIONAL ENVIRONMENT de Francisco Jorgan Cabral Clarindo e Paulo Henrique Garcia Mansur; DISTANCE LEARNING: A PANORAMA OF ITS EXPANSION IN NORTHERN BRAZIL de Jonas Fernando Petry, Gustavo da Rosa Borges e Maria José Carvalho de Souza Domingues; WEB 2.0 TOOLS AS A PEDAGOGICAL RESOURCE de Ivaldir Farias Junior, Marcelo Mendonça Texeira, Dennis Sávio, Tiago Ferreira e Colaboradores: Micaías Paiva e Lenildo Nogueira; DEVELOPMENT OF A COMPREHENSIVE SUPPORT OF TECHNOLOGICAL INNOVATION TO ALLOW THE USE OF QUALITY MANAGEMENT IN INSTITUTIONS OF HIGHER EDUCATION de Cesar Enrique Estrada Gutiérrez, Eva Martha Chaparro Salinas, Maria de La Luz Sánchez Paz e Felisa Yaerim López Botello; CURRICULUM MATRIX PROFILE OF COURSES IN THE AREA OF CONTROLLING de Evandro de Nez, Sheila Jeane Schulz e Carlos Eduardo Facin Lavarda. Segundo a moderadora, a sessão ocorreu pontualmente. As temáticas trabalhadas foram bastante exploradas nos debates e houve pleno envolvimento dos ouvintes em todas as apresentações. A sessão 4(F) com o tema HEA – Healthcare Information Systems and Technology, foi moderada por Liliana Giusti Serra e contou com os trabalhos: STRATEGIES OPERATIONS HEALTH SERVICES PROCESSES IN ORGANIZATIONAL RESTRUCTURING: IMPACT ON MARKETING ORGANIZATIONS HEALTH SERVICES AND VALIDATION OF WORD OF MOUTH MARKETING de Sonia Francisca Monken e Thaissa Souza Mendes; PROFILE OF FINANCIAL MANAGEMENT IN APPLIED INDUSTRIAL COMPANIES SERVED BY CORE

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

497

PEIEX SÃO PAULO / SP BETWEEN 2010-2012 de José Flávio Messias, José Ultemar da Silva e Alessandro Marco Rosini; THE NATIONAL HEALTH CARD IN SUPPORT OF MARKETING ACTIONS IN HEALTH de Sonia Francisca Monken, Ernesto Stangueti e Thaissa Souza Mendes; UBISALUD: CHANGE LIFESTYLE RECOMMENDATION SYSTEMS IN PREVENTING NON-COMMUNICABLE DISEASES de Concepción Pérez de Celis, María J. Somodevilla, Ivo H. Pineda, Alejandro Serrano e Colaboradores: J. Paul Márquez, Alejandro Herrera e Karen Ramírez; CHOOSING A ERP: IN A HOSPITAL ENVIRONMENT de Geraldo Lima, Alessandro Marco Rosini, Kênia Bianchi, Carla Celso L. Souza e Colaboradores: Cauê R. Schuster E Rogério Oliveira. Segundo a moderadora, a sessão reuniu temática na área da saúde, abordando o uso de tecnologia da informação como instrumento auxiliar na identificação de problemas. As pesquisas foram relevantes, com abordagens distintas e complementares, suscitando a discussão entre os demais ouvintes. Na sessão 4(G), cujo tema foi ITM – Information Technology Management, a moderação foi realizada por Alberto Medeiros Junior, que coordenou a apresentação dos trabalhos: AN ANALYSIS OF THE CLOUD COMPUTING APPROACH TO SOA de Tâmara Batista Reis e Colaborador: Paulo Caetano da Silva; CRITERIA SELECTION FOR NOSQL DBMS: AN APPLICATION OF THE DELPHI TECHNIQUE de Alexandre Morais de Souza, Edmir Parada Vasques Prado, Sarajane Marques Peres e Violeta Sun; METRICS OF THE PROFESSIONAL CAREER PLAN LINKED TO QUALITY OF PROCESS OF SOFTWARE DEVELOPMENT AT IN INFORMATION TECHNOLOGY COMPANIES – PCTI de Edilaine Rodrigues Soares, Fernando Hadad Zaidan e George Leal Jamil. Segundo o moderador, a sessão ocorreu dentro dos padrões previstos, sendo as temáticas discutidas após as apresentações através da elaboração de questões com enfoque no tema de computação em nuvem e SGBD NOSQL. O trabalho STRATEGY PLANNING COMMUNITY OF PRACTICE: CASE STUDY OF SMALL AND MEDIUM-SIZED HIGH TECHNOLOGY ENTERPRISES de Taciana de Barros Jerônimo e Raniere Rodrigues dos Santos não foi apresentado devido à ausência dos autores.

A sessão 4(H), com o tema COMM-Research Communication, teve como moderadora Nelma Zubek Valente e contou com a apresentação dos trabalhos: THE INFLUENCE OF STAKEHOLDERS ON THE PMBOK KNOWLEDGE AREAS: THE CASE OF A PHARMACEUTICAL INDUSTRY de Fernando Moura Palhares, Anselmo Battisti e Maria Alexandra Cunha; BALANCED SCORECARD SOFTWARES: BRAZILIAN MANAGERS PERCEPTION de Cristiana Fernandes de Muylder, Cristiano Ledo B. Cruz e André Miquelão Zandim Guedes; WEAK SIGNALS MANAGEMENT, ENTREPRENEURSHIP AND UNCERTAINTY: A RELATIONAL THEORETICAL ESSAY UNDER THE PERSPECTIVE OF INTELLIGENCE de Raquel JanissekMuniz e Mery Rose de Mello Blanck; TEORIAS CRÍTICAS DE CURRÍCULO: CONTRIBUIÇÕES PRELIMINARES PARA REPENSAR A FORMAÇÃO NAS CIÊNCIAS DA INFORMAÇÃO de Marielle Barros de Moraes e Ariel Sánchez Espinoza; SOCIAL NETWORK ANALYSIS AS A TOOL TO SUPPORT IN COMPUTER COURSES de Elton Rabelo, Thiago Magela Dias, Patrícia Dias e Gray Moita. Segundo a moderadora, os trabalhos apresentados geraram bastante interesse nos

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

498

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

participantes da sessão que levantaram relevantes questões, comentários, sugestões e observações muito construtivas para o avanço das pesquisas apresentadas. O primeiro trabalho apresentado levantou questões e comentários sobre a abordagem multi-método adotada e recebeu sugestões para a aplicação do framework proposto, incluindo um cruzamento entre gestão estratégica e vantagem competitiva; questões e comentários sobre o instrumento de coleta de dados assim como sobre o tipo de organizações onde a pesquisa será aplicada especialmente no quesito “orientação empreendedora”. Reflexões foram também levantadas sobre as possíveis dificuldades de entendimento do instrumento de coleta de dados por parte de pequenos empreendedores e ainda se os dados qualitativos levantados pelo instrumento gerariam informações que levariam a dados quantitativos. O segundo trabalho apresentado recebeu comentários em relação às diferenças existentes entre Metodologia BSC e Software BSC e por isso o cuidado na escolha da terminologia adequada. Questionou-se também sobre a amostra das empresas estudadas no tocante ao estágio da adoção do BSC, se na fase inicial ou avançada e também sobre o corte da amostra. O terceiro trabalho apresentado foi questionado quanto às razões da escolha da base bibliográfica utilizada. Sugeriu-se também a utilização dos fundamentos do Curríiculo Mundial. No quarto trabalho apresentado, questionou-se sobre a adequação da tipologia da pesquisa, uma vez que um dos pesquisadores faz parte da empresa pesquisada.

MAY 30 / 30 DE MAIO (FRIDAY / SEXTA-FEIRA) Na sessão 5(A), com o tema AIS RF - Accounting and Enterprise Information Systems, a moderação foi realizada pela Profa. Vilma Geni Slomski e foram apresentados os seguintes trabalhos: DECISION FOR MORE GREAT OPTION FOR SALE PRICE THROUGH GAME THEORY: A CASE STUDY WITH IMPLEMENTATION OF PURE STRATEGY IN A HOTEL COMPANY de Alini da Silva, Caroline Sulzbach Pletsch, Nelson Hein, Jorge Eduardo Scarpin; APPLICATION OF THE FACTOR ANALYSIS ON DATA MINING OF THE PERFORMANCE SOCIAL TO COMPANIES THE SECTOR CONSUMPTION CYCLIC LISTED ON BM&FBOVESPA de Nelson Hein, Fernanda Kreuzberg, Larissa Degenhart, Mara Vogt, Colaborador Moacir Manoel Rodrigues Junior; OCCASIONED CHANGES IN INFORMATION SYSTEMS IN ACCOUNTING ACCOUNTING OFFICE de Rodrigo Barraco Marassi, Vania Regina Morás, Jaqueline Carla Guse, Fabricia Silva da Rosa. Segundo a moderadora, o primeiro trabalho refere-se às Práticas de Gestão de Custo. O segundo trabalho, ainda de acordo com a moderadora, segue o rigor metodológico e atende a padrões de qualidade e de confiabilidade. O terceiro trabalho refere-se à Prática de Gestão dos Sistemas de Informação e, para a moderadora, o método de análise deveria ser descrito. Na sessão 5(B), com o tema ECOM - E-business and E-commerce, a moderação foi realizada pela Profa. Maria Irene da Fonseca e foram apresentados os seguintes trabalhos: E-BUSINESS ADOPTION BY SMALL AND MEDIUM-SIZED BRAZILIAN ENTERPRISES de Érica Siqueira, Vladimir Chagas, Cesar Alexandre

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

499

de Souza, Edmir Parada Vasques Prado; AN EXPLORATORY ANALYSIS OF THE GROWTH OF THE BRAZILIAN BANKING CYBERSPACE de Fabio Bergamo, Jarson Santos Dantas, Jó Santos da Silva, Itiel Moraes da Silva e o colaborador Ivo Gonzalez Junior. Segundo a moderadora, as palestras abordaram temas atuais pertinentes ao evento e à área. As discussões visaram, principalmente, contribuir para o desenvolvimento futuro das pesquisas apresentadas. Informou também que as palestras foram apresentadas em sua totalidade. Na sessão 5(C), com o tema EDU RF II - IS and IT Education and Curriculum Development, a moderação foi realizada pelo Prof. Marcos Ricardo Rosa George e foram apresentados os seguintes trabalhos: THE CONTRIBUTION OF INFORMATION TECHNOLOGY IN EDUCATION: A BIBLIOMETRIC STUDY OF THE APPLICATION OF SIMULATION GAMES IN OPERATIONS MANAGEMENT (OM) de Claudia Aparecida de Mattos; ESTABLISHMENT OF MICROBLOGS FOR COMMUNICATION AND INTERACTION IN MOBILE LEARNING ENVIRONMENTS de Nemésio Duarte Freitas Filho, Sofia Larissa da Costa Paiva, Draylson Micael de Souza, Ellen Francine Barbosa; FREE EDUCATIONAL SOFTWARE AND PROSPECTS OF INTERDISCIPLINARITY de Alessandro Marco Rosini, Aurélio Vitorino, Cezar Amaral da Silva. De acordo com o moderador, as discussões apresentaram bom nível, especialmente no uso de jogos e metodologias ativas de ensino. O mesmo sugeriu promover o debate no uso de tecnologias móveis em sala de aula e também reforçou a importância de ter acesso aos artigos antes da sessão. Na sessão 5(D), com o tema EGOV III - E-Governance and Public Policies, a moderação foi realizada pela Profa. Claudia Maria da Costa Archer e foram apresentados os seguintes trabalhos: PROMOTING FISCAL CITIZENSHIP THROUGH E-TAXATION de Rute Abreu, Fátima David, Liliane Cristina Segura e Henrique Formigoni; COLOMBIAN ELETRONIC INVOICE: ANALYSIS OF THE OPERATIONAL MODEL de Sérgio Roberto da Silva, Ludivia Hernandez Aros, Márcia Welita da Silva, Laura Constanza Gallego Cossio; A PROPOSAL OF PUBLIC BUDGET DATA PUBLISHING ON THE WEB de Marcelo Tavares de Santana, Gisele da Silva Craveiro. A moderadora informou que a segunda apresentação despertou interesse e que os resultados da discussão contaram com a contribuição da platéia. De modo geral, as discussões após as apresentações, contribuíram para aprofundar o conhecimento sobre os temas. Na sessão 5(E), com o tema ISM RF - Information Systems Management, a moderação foi realizada pelo Prof. Renato Ribeiro Nogueira Ferraz e foram apresentados os seguintes trabalhos: ERP SYSTEM IMPLEMENTATION WITH BUSINESS PROCESS MANAGEMENT AT PETRÓPOLIS GROUP de Alaercio Nicoletti Junior e Fernando Celso de Campos; USER PARTICIPATION IN THE DEVELOPMENT OF INFORMATION SYSTEMS: SOME IMPRESSIONS FROM PROFESSIONAL DEVELOPERS OF SPECIALIZED CLUSTERS de Jairo Simião Dornelas, Rodrigo Cesar Reis de Oliveira e Américo Nobre Amorim; A COMPARATIVE STUDY OF PROJECT PORTFOLIOS MANAGEMENT MATURITY MODEL de Marco Alexandre Terlizzi, Heverton Roberto Oliveira Cesar de Moraes e Marcos Paixão Garcez. Segundo o moderador, os temas eram importantes

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

500

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

e mereciam mais tempo para discussão. De modo geral, as sessões ocorreram pontualmente com tempo para apresentação e discussão de todos os trabalhos. Na sessão 5(F), com o tema ITM RF - II Information Technology Management, a moderação foi realizada pelo Prof. Gilberto Perez e foram apresentados os seguintes trabalhos: DIAGNOSIS AND TREATMENT OF INCIDENTS IN THE COMPUTER NETWORK de Almir Carlos da Silva e Eric A. M. Fagotto; STRATEGIC ALIGNMENT BETWEEN BUSINESS AND INFORMATION TECHNOLOGY: A BIBLIOMETRIC STUDY OF BRAZILIAN CONFERENCES IN THE PERIOD 2009-2013 de Laércio Moura de Almeida e Ernani Marques dos Santos; IT BUDGET AND USER SATISFACTION: A STUDY CASE OF A CAR DEALERSHIPS GROUP IN NATAL-RN de Hudson Soares Leite Junior e Marcos Fernando Machado de Medeiros. Segundo o moderador, houve pontualidade nas apresentações, as temáticas abordadas foram eficientes e as discussões, entre as cinco pessoas presentes, foram muito proveitosas principalmente devido ao reduzido número de trabalhos desta sessão (3). Na sessão 5(G), com o tema ITM III - Information Technology Management, a moderação foi realizada pelo Prof. João Carlos Hipólito e foram apresentados os seguintes trabalhos: MEASURING TECHNOLOGICAL MATURITY LEVELS WITH DATA MINING de Rafael Herden Campos, Rocio Rocha, Angel Cobo, Adolfo Alberto Vanti; ANALYSIS OF THE RELATIONSHIP BETWEEN CORPORATE GOVERNANCE AND INFORMATION TECHNOLOGY GOVERNANCE IN BRAZILIAN ORGANIZATIONS de Andréa P. Gonçalves, Marcia R. Sales e Hamilton Luiz Correa; INTERACTION AND EFFECTS OF INTERNAL AND EXTERNAL IT CAPABILITIES ON PERFORMANCE: THEORETICAL AND EMPIRICAL PERSPECTIVES de Deyvison de Lima Oliveira, Antonio Carlos Gastaud Maçada e Gessy Dhein Oliveira. Esta sessão contou com apenas três apresentações o que contribuiu para fomentar o debate, uma vez que houve maior disponibilidade de tempo. O moderador também informou que o agrupamento e a concentração de artigos de mesma temática foram importantes por fomentar o debate, o que levou à maturação e complexidade das questões e contribuições abordadas. A sessão ocorreu pontualmente, conforme estabelecido. A sessão 6(A), com o tema AIS – Accounting and Enterprise Information Systems, teve como moderador Rodrigo Barraco Marassi e contou com a apresentação dos trabalhos: CULTURE AND ACCOUNTING: THE CHALLENGE OF CONVERGENCE IN THE PUBLIC SECTOR IN BRAZIL de Silvana Karina de Melo Travassos e José Isidio de Freitas Costa; CASH BALANCE PREDICTION: A COMPARISON BETWEEN ARTIFICIAL NEURAL NETWORKS AND LINEAR REGRESSIONS de Liza Satie Eguchi e Marcelo Botelho da Costa Moraes ; THE ENVIRONMENT IN THE CONTROLLING ENTITIES COOPERATIVE de Fernando Maciel Ramos, Vanderlei Gollo e Carlos Eduardo Facin Lavarda; TARGET COSTING UNDER THE THEORY APPROACH IN AGRICULTURAL ORGANIZATIONS CONTINGENCY de Maria Margarete Baccin Brizolla, Tânia Cristina Chiarello, Luiza Betânia Fasolin e Fabricia Silva da Rosa. Segundo o moderador, as discussões de todos os trabalhos fluíram bem. Foram levantadas questões de caráter metodológico e principalmente sobre os ramos empresariais dos estudos realizados.

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

501

A sessão 6(B), com o tema ICT – ICT4DEV – Information and Communication Technologies for Development and Assistive Technologies, teve como moderador Maria Irene da Fonseca e Sá e contou com a apresentação dos trabalhos: THE USE OF SCRIPTLATTES TOOL FOR EXTRACTION AND ON LINE AVAILABILITY OF ACADEMIC PRODUCTION FROM A DEPARTMENT OF STRICTO SENSU IN MANAGEMENT de Renato Ribeiro Nogueira Ferraz, Luc Marie Quoniam e Emerson Antônio Maccari; THE SCIENTIFIC JOURNAL PERSPECTIVAS EM GESTÃO & CONHECIMENTO (PG&C): HISTORY OF IDEALIZATION AND EXPERIENCE OF EDITORIAL MANAGEMENT de Luciana Ferreira da Costa, Alan Curcino Pedreira da Silva, Emeide Nóbrega Duarte e Jorge da Oliveira Gomes; DIGITAL LIBRARIES: A STUDY ON THE CALL OF THE DIVERSITY OF USER NEEDS de Maria Irene da Fonseca e Sá e Rosali Fernandez de Souza. Segundo o moderador, os temas apresentados foram pertinentes à área e as discussões formuladas foram muito ricas. O trabalho ANALYSIS OF THE RELATIONSHIP BETWEEN CORPORATE GOVERNANCE AND INFORMATION TECHNOLOGY GOVERNANCE IN BRAZILIAN ORGANIZATIONS de Andréa P. Gonçalves, Marcia R. Sales e Hamilton Luiz Correa não foi apresentado, pois já havia sido integrado à sessão 5G. Na sessão 6(C), o tema ICT - ICT4DEV – Information and Communication Technologies for Development and Assistive Technologies, teve a moderação de Rafael Herden Campos e contou com a apresentação dos trabalhos: ENTERPRISE CONTENT MANAGEMENT de Francisco Carlos Paletta e Daniel Dias; THE DEVELOPMENT OF WEB 3.0: LINKED DATA AND DBPEDIA de Marcos Luiz Mucheroni e Francisco Carlos Paletta; AN OVERVIEW ABOUT ENVIRONMENTAL SCANNING MODELS ADOPTED BY COMPETITIVE INTELLIGENCE PROGRAMS IN BRAZIL de Olavo Viana Cabral Netto e Fernando José Barbin Laurindo; INFORMATION GOVERNANCE: TOWARDS A CONCEPTUAL FRAMEWORK FOR THE INTEGRATIVE MANAGEMENT OF INFORMATION AND KNOWLEDGE IN ORGANIZATIONS de Patrícia Rocha Bello Bertin. Segundo a moderadora, os temas foram atuais com muitas discussões pertinentes e indutivas, atingindo um nível de excelência. A sessão 6(D), com o tema ESD – Engineering and Software Development, teve como moderador Marco Alexandre Terlizzi e contou com a apresentação dos trabalhos: SYSTEM SUPPORT TO MUNICIPALITIES – SIAPREF de Roney Diego Queiroz Santos, Cristiano de Sousa Gutierrez e Renata dos Santos Luz de Oliveira; MANAGEMENT BY PERFORMANCE INDICATORS IN MALLS de Leonardo Fabris Lugoboni, Flavio Hourneaux Junior, Veronice Francisca da Costa Ferreira e Luis Paulo Cristino Mota. Segundo o moderador, o nível das discussões foi bastante alto. As questões levantadas tiveram o objetivo de auxiliar na melhoria dos trabalhos, enfocando temas como sucesso em gestão de projetos, gestão dos processos de desenvolvimento de sistemas de informação. O artigo ADOPTING AGILE SOFTWARE DEVELOPMENT WITH EMPHASIS ON QUALITY PROJECT de Diego Noronha, Elton Rabelo, Thiago Magela Rodrigues Dias, Patrícia Mascarenhas Dias e Gray Farias Moita não foi apresentado devido à ausência dos autores. Na sessão 6(E), o tema HEA – Healthcare Information Systems and Technology, foi

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

502

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

moderado por Alessandro Marco Rosini e contou com a apresentação dos trabalhos: MANAGEMENT HEALTH SYSTEM OF THE CITY OF SÃO PAULO BASED ON ASSESSMENT STANDARDS OF PMAQ-AB: AN CASE STUDY ON REGION OF CIDADE TIRADENTES de Marília Bezulle Chimara e Edinho Silva; IMPROVEMENT ASSISTANCE STRATEGY INVOLVING FAMILY/CAREGIVER IN PREVENTING ADVERSE EVENTS IN HOSPITALIZED PATIENTS de Lígia Maria Quitério e Márcia Cristina Zago Novaretti; DIAGNOSIS OF INFORMATION AND TECHNOLOGY MANAGEMENT SYSTEMS IN THE HOSPITALS OF SÃO LUÍS – MA de Will Ribamar Mendes Almeida, Cláudia Maria da Costa Archer, José Antônio Fecury Ferreira e André Rossanno Mendes Almeida. Segundo o moderador, a sessão ocorreu dentro os padrões previstos no programa. Após as apresentações, houve debate sobre as temáticas aboradas. A sessão 6(F), tratou do tema ISM – Information Systems Management, teve como moderador Jairo Simião Dornelas e contou com a apresentação dos trabalhos: COMPENSATION FOR STOCK OPTIONS AND FINANCIAL PERFORMANCE: AN ANALYSIS OF BRAZILIAN PUBLIC COMPANIES de Samuel de Oliveira Durso, Danielle Penido Teixeira, Marcia Athayde Moreira e Ana Carolina Vasconcelos Colares; THERE IS A CENTER OF RESEARCH ON INFORMATION SYSTEMS IN BRAZIL? AN ANALYSIS OF THE SCIENTIFIC PRODUCTION FROM THE BIBLIOMETRIC STUDY OF THE JOURNAL OF INFORMATION SYSTEMS AND TECHNOLOGY MANAGEMENT de Samuel de Oliveira Durso e Patrícia Antonacci Neves; ANALYSIS OF FACTORS ASSOCIATED WITH THE ACCEPTANCE AND REJECTION OF THE USERS OF AN ERP SYSTEM de Claudia Aparecida de Mattos e Colaborador: Fernando José Barbin Laurindo; MECHANISMS OF NONCOMPLIANCE OF ERP ORGANIZATIONS: SUGGESTED FRAMEWORK de Sérgio Roberto da Silva, Márcia Welita da Silva, Antonio Carlos de Alcantara Thimóteo e Marcelo Rabelo Henrique; EVALUATION OF THE POTENTIAL USE OF BALANCED SCORECARD IN THE SMALL AND MEDIUM COMPANIES de Sérgio Lins Pessoa, José Alfredo Ferreira Costa e Mariana Rodrigues de Almeida; EFFECTIVE IMPLEMENTATION OF STRATEGIC PLANNING IN THE COURTS OF RIO GRANDE DO NORTE de José André Diogo Neto e José Alfredo Ferreira Costa. Segundo o moderador, as discussões atingiram níveis bons de aprofundamento e as temáticas foram abordadas de modo significativo. Na sessão 6(G), KGM – Knowledge Management and Business Intelligence , a moderação foi realizada por Itiel Moraes da Silva e contou com a apresentação dos trabalhos: FUNCTIONAL ILLITERACY: AN EMPIRICAL ANALYSIS TO UNDERGRADUATE ACCOUNTING OF RONDÔNIA de João Carlos Hipólito Bernardes do Nascimento, Adriano Marcos, Marcela de Almeida Lima, Eliana Maria Pereira e Colaborador: Juliana Reis Bernardes; NEW PERSPECTIVES USING BIG DATA: A STUDY OF BIBLIOMETRIC 2000-2012 de Ivan Menerval da Silva e Fernando Celso de Campos; THE PERCEPTION OF STUDENTS ABOUT THE LEARNING IN THE DISTANCE EDUCATION IN ACCOUNTING SCIENCES: A STUDY IN INSTITUTIONS GRADUATION JUAZEIRO - BA AND PETROLINA – PE de Juliana Rogéria da Costa, Wellington Dantas de Sousa, João Carlos Hipólito Bernardes do Nascimento, Juliana Reis Bernardes e Colaborador: Iris Ribeiro de Melo; APPLICATION OF THE ORGANIZATIONAL KNOWLEDGE ASSESSMENT

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

503

METHOD IN THE DIAGNOSIS OF KNOWLEDGE MANAGEMENT IN A BRANCH OF A MULTINATIONAL COMPANY OF INFORMATION TECHNOLOGY de Vanessa Costa Silva e Adriane Maria Arantes de Carvalho. Segundo o moderador, a sessão consistiu basicamente na temática aprendizagem organizacional e educacional, juntamente com a gestão do conhecimento. O nível das discussões foi satisfatório para o objetivo do congresso. As questões levantadas pelos participantes se concentraram nas dificuldades e fragilidades do referencial teórico e metodológico das pesquisas. As temáticas debatidas surgiram de forma relevante para o contexto educacional contemporâneo, abrindo novas possibilidades de aprofundamento das pesquisas das temáticas apresentadas de cada área. A sessão 6(H), com o tema PRM - Project Management, moderada por Érica Souza Siqueira, contou com a apresentação dos trabalhos: ELETRONIC INVOICE IN BRAZIL: ANALYSIS OF THE OPERATIONAL MODEL de Sergio Roberto Silva, Rosangela de Carvalho e Ludivia Hernandes A.; MULTILEVEL MEASUREMENT SYSTEM PERFORMANCE TO ESTABLISH STRATEGIC DIALOGUE: A CASE STUDY IN ACCOUNTING SERVICES de André Gobette Santana, Silvio Aparecido Teixeira e Carlos Facin Lavarda.

G) TRABALHOS QUE RECEBERAM MENÇÃO HONROSA NO CONTECSI-2014:

11º

O 11º CONTECSI também se preocupou em destacar os melhores trabalhos apresentados no evento, tanto do ponto de vista acadêmico quanto prático. Para tanto, foram levados em consideração os pareceres dos membros do comitê científico do evento para a classificação dos três melhores trabalhos. Assim, receberam menção honrosa, os seguintes autores com os respectivos trabalhos: Best Paper Award - 11th CONTECSI / Melhores trabalhos do CONTECSI indicado pelos avaliadores: ANÁLISE DOS CONTRATOS DE TERCEIRIZAÇÃO DE SERVIÇOS DE TI NO TCU E DA LEGISLAÇÃO QUE REGE ESTAS CONTRATAÇÕES NA ADMINISTRAÇÃO PÚBLICA FEDERAL Graziela Ferreira Guarda Edgard Costa Oliveira Rafael Timóteo de Sousa Junior PROPOSIÇÃO DE UM PORTAL DE EGRESSO (ALUMNI) BASEADO EM BENCHMARKING E PROCESSO INOVADOR Proposition of an alumni portal based on benchmarking and innovative process Gislaine Cristina dos Santos Teixeira Emerson Antonio Maccari GESTÃO DE SEGURANÇA DA INFORMAÇÃO: ESTUDO DE CASO EM UMA INSTITUIÇÃO FINANCEIRA

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

504

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

Antonio Maria da Silva Carpes Elisandra Henn Diel Francisco Carlos Fernandes TOPICS AIS AUD ECOM EDU EGOV ESD HEA ICT INT INV ISM ITM KMG PRM SEC SOC XBRL COMM DOCT MST POST

Accounting and Enterprise Information Systems / Sistemas de Informações Contábeis e Empresariais Systems Auditing and IT Governance / Auditoria de Sistemas e Governança em TI E-business and E-commerce / Comércio Eletrônico IS and IT Education and Curriculum Development/Educação e Curriculo em SI e TI E-governance and Public Policies / Governo Eletrônico e Políticas Públicas Engineering and Software Development / Engenharia e Desenvolvimento de Software Healthcare Information Systems and Technology / Sistemas de Informação em Saúde e Tecnologia ICT4DEV-Information and Communication Technologies for Development and Assistive Technologies / TICS para Desenvolvimento Internet and Virtual Communication and Social Networks / Internet e Comunidades Virtuais e Redes Sociais IS and IT Innovation and Change / Inovações e mudanças em SI e TI Information Systems Management / Gestão de Sistemas de Informação Information Technology Management/ Gestão de Tecnologia de Informação Knowledge Management and Business Intelligence / Gestão do Conhecimento e BI Project Management / Gestão de Projetos IS and IT Security / Segurança em SI e TI Social Issues in IS and TI / Questões Sociais em SI e TI XBRL and Enterprise Onthology / XBRL e Ontologia empresarial Research Communication / Comunicação de Pesquisa Doctoral Consortium / Consórcio Doutoral Master Colloquium / Coloquio de Mestrado Postdoctoral Consortium / Consórcio Pós-doutoral

PS - Parallel Session/Sessão Paralela RF - Research Forum/Fórum de Pesquisa ENG - Session in English / Sessão em Inglês

UNIVERSITY OF SAO PAULO UNIVERSIDADE DE SÃO PAULO Prof. Dr. Marco Antonio Zago Rector | Reitor Prof. Dr. Vahan Agopyan Vice-Rector | Vice-Reitor

SCHOOL OF ECONOMICS, BUSINESS AND ACCOUNTANCY FACULDADE DE ECONOMIA, ADMINISTRAÇÃO E CONTABILIDADE Prof. Dr. Reinaldo Guerreiro Diretor | Dean

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

505

Prof. Dr. Nicolau Reinhard Vice-Dean | Vice-Diretor Prof.Dr. Edgard Bruno Cornachione Jr. Head of Accountancy and Actuary Department | Chefe do Departamento de Contabilidade e Atuária Prof. Dr. Edson Luiz Riccio President of CCInt - International Cooperation Office - FEA USP TECSI Director - University of São Paulo Organization | Organização FEA USP - Universidade de São Paulo TECSI – Information Systems and Technology Management Lab CCInt FEA

Support | Apoio FEA USP - Universidade de São Paulo CAPES – Coordenação de Aperfeiçoamento de Pessoal de Nível Superior CNPq - Conselho Nacional de Desenvolvimento Científico e Tecnológico FAPESP – Fundação de Amparo à Pesquisa do Estado de São Paulo UNINOVE – Universidade Nove de Julho PRCEU - Pró-reitoria de Cultura e Extensão Universitária Ministério da Ciência e Tecnologia CaseWare Analytics Partners | Parceiros CARLAB - Rutgers Business School, The State Uiversity of New Jersey, USA CETAC – Centre for Studies in Technologies, Arts and Communication Sciences, Porto Univ., Portugal AIS – Association for Information Systems ANPAD – Associação Nacional de Pós-Graduação e Pesquisa em Administração São Paulo Convention & Visitors Bureau ANEFAC – Associação Nacional dos Executivos de Finanças, Administração e Contabilidade ISACA – Audit and Control Association FIPECAFI – Fundações Instituto de Pesquisas Contábeis, Atuariais e Financeiras GESITI Centro de Tecnologia de Informação Renato Archer University of Insubria, Italy

SCIENTIFIC COMMITTEE| COMITÊ CIENTÍFICO A. A. de Souza, Federal University of Minas Gerais, Brazil A. A. Vanti, University of Vale do Rio dos Sinos, Brazil A. Carlos dos Santos, Federal University of São Carlos, Brazil A. de Medeiros Júnior, Mackenzie Presbyterian University, Brazil A. Fischmann, University of São Paulo, Brazil A. J. Balloni, Centro de Pesquisa Renato Archer, Brazil A. Malheiro da Silva, Universidade do Minho, Portugal A. Rosini, Catholic University of São Paulo, Brazil A. Sangster, Univesity of Middlesex, UK B. Quinio, Paris Quest, Nanterre, France C. A. Biancolino, Nove de Julho University, Brazil C. Benavant, Paris Quest, Nanterre, France C. D. Prá Martens, Pierre Mendès France University, France

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

506

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

C. D. S. Miranda, University of São Paulo, Brazil C. D. Santos Jr, University of São Paulo, Brazil D. A. Rezende, Catholic University of Paraná, Brazil C. Piaggesi, Fondazione Rosselli Americas, USA E. Brusseau, Paris Dauphine, France E. L. Riccio, University of São Paulo, Brazil E. M. Chaparro Salinas, Universidad Autónoma del Estado del México, México E. M. Luciano, Catholic University of Rio Grandedo Sul, Brazil E. M. R. Estrada, Universidad Autónoma del Estado del México, México E. Maccari, Nove de Julho University, Brazil F. Colmenero Ferreira, University of Madeira, Portugal F. J. Laurindo, University of São Paulo, Brazil G. L. Jamil, Education and Culture Foundation of Minas Gerais, Brazil G. Perez, Mackenzie Presbyterian University, Brazil G. Schwartz, University of São Paulo, Brazil H. Freitas, Federal University of Rio Grande do Sul, Brazil H. N. Rito Ribeiro, College of Technology and Management, Portugal I. Custódio, University of São Paulo, Brazil J. A. Botello, Universidad Autónoma del Estado del México, México J. A. F. Costa, Federal University of Rio Grande do Norte, Brazil J. D. Oliveira Neto, University of São Paulo/RP, Brazil J. G. D. A. Teixeira Filho, Federal University of Pernambuco, Brazil J. O. De Sordi, Catholic University of Santos, Brazil J. O. Imoniana, Mackenzie Presbyterian University, Brazil J. P. Alcázar, University of São Paulo, Brazil J. Pimenta Matos, Federal Institute of Education Science and Technology os São Paulo, Brazil J. Rodrigues Filho, Federal University of Paraiba, Brazil L. C. Rodrigues, Nove de Julho University, Brazil L. de Faria Lopes, Federal University of São Carlos, Brazil L. Quoniam, University of Toulon-Var, France M. Alles, State University of New Jersey, USA M. A. Gouvêa, University of São Paulo, Brazil M. A. Hirose Fedichina, Centro Universitário de Jales, Brazil M. A. Silveira, Centro de Technologia da Informação Renato Archer, Brazil M. C. Machado, Instituto Tecnológico da Aeronáutica, Brazil M. G. Sakata, University of São Paulo, Brazil M. J. Bacic, University of Campinas, Brazil M. N. Bessagnet, Universitè de Pau, France M. R. S. Peters, Armando Álvares Penteado Foundation, Brazil M. Vasarhelyi, Rutgers University, USA N. Azoury, Holy Spirit University of Kraslik, Lebanon N. Galegale, University of São Paulo, Brazil N. Reinhard, University of São Paulo, Brazil O. R. de Mendonça Neto, Mackenzie Presbyterian University, Brazil P. C. da Silva, UNIFACS, Brazil R. C. Penteado Filho, Empresa Brasileira de Pesquisa Agropecuária, Brazil R. Ferreira, Federal University of Mato Grosso, Brazil

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

507

R. G. Gonçalves, University of São Paulo/RP, Brazil R. L. Cardoso, Mackenzie Presbyterian University, Brazil R. M. C. Figueiredo, University of Brasilia, Brazil R. Pacheco da Costa, University of São Paulo, Brazil S. A. dos Santos, University of São Paulo, Brazil S. R. P. Alves, Instituto Politécnico de Leiria, Portugal T. A. Peters Filho, Faculdade de Engenharia Industrial, Brazil V. Branco de Holanda, Federal University of Rio Grande do Norte, Brazil V. Slomski, University of São Paulo, Brazil W. Castelnovo, University Dell’Insubria, Italy

COORDINATOR | COORDENAÇÃO Prof. Dr. Edson Luiz Riccio President of CCInt FEA USP –TECSI Director University of São Paulo ORGANIZING COMMITTEE | COMITÊ ORGANIZADOR Prof. Dr. Edson Luiz Riccio President of CCInt FEA USP –TECSI Director University of São Paulo Prof. Dr. Miklos Vasarhelyi Director of Rutgers Accounting Center – Rutgers Business School – The State University of New Jersey, USA Prof. Dr. Luc Quoniam Aix-Marseille Université / Uninove, Brasil Prof. Dr. Armando Malheiro Silva CETAC- Centre for Studies in Technologies, Arts and Communication Sciences Porto University, Portugal Prof. Dr. Christophe Benavent CEROS - Centre d’études et de recherches sur les organisations et sur les stratégies Université Paris Ouest Nanterre, France Prof. Dr. Jae Jon Kim Chonnam National University, South Korea INVITED KEYNOTE SPEAKER Profa. Dra. Jane Fedorowicz Bentley University,USA Prof. Dr. Jean Claude Guedon Université de Montréal, Canadá

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

508

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

Prof. Dr. Walter Castelnovo University of Insubria, Italy Prof. Dr. Armando Malheiro – CHAIR ICT PANEL Universidade do Porto, Portugal Prof. Dr. João Álvaro Carvalho Universidade do Minho, Portugal Prof. Dr. Miklos Vasarhelyi CHAIR – 31st WCARS Rutgers University,USA Profa. Deniz Applebaum | 31st WCARS Rutgers University, USA Profa. Dra. Helen Brown | 31st WCARS Rutgers University, USA Prof. Dr. Kevin Mofitt | 31st WCARS University of Kansas, USA SUPPORT ORGANIZATION COMMITTEE| COMITÊ ORGANIZADOR DE APOIO Profa. Dra. Marici Gramacho Sakata TECSI FEA USP Researcher, Brazil Prof. Dr. George Leal Jamil Doctoral Consortium and Mater Colloquium Organizer – Fumec University, Brazil Prof. Dr. Washington Lopes da Silva 31st WCARS World Continuous Auditing and Reporting Systems Symposium Prof. Victor Freitas de Azevedo Barros Algoritmi Centre/ University of Minho, Portugal Prof.Msc.Denis Alves e Lima Poster Section Organizer – FEA USP Profa. Dra. Nelma Terezinha Zubek Valente State University of Ponta Grossa, Paraná, Brazil/ Equipe TECSI FEA/ USP Henrique Schulz and Daniel Espana Valle Webmaster Msc. Ligia Capobianco Mariana Coelho Naldi

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

Resultados do 11º. CONTECSI USP - Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação /International USP Conference on Information Systems and Technology Managament

509

Agnes Massuyama Thayane Arruda Equipe TECSI/FEA/USP

AUTORES O 11º CONTECSI dispôs de um espaço no stand para anunciar o lançamento de livros dos autores presentes, relativos aos assuntos relacionados ao evento. Assim, todos os autores puderam divulgar e lançar seus livros no congresso. O espaço em questão não foi destinado à venda, mas à divulgação das referidas obras. EVENTOS CULTURAIS MÚSICA NO 11º CONTECSI

Apresentação de Encerramento – 14 de Junho 

Virado na Gameleira o Mariana Corado, Aline Reis e Igor Caracas

Noticia publicada no Portal da FEA USP

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

510

Riccio, E.l., Sakata, M. C., Valente, N. T. Z., Capobianco, L.

11º CONTECSI - CONGRESSO INTERNACIONAL DE GESTÃO DA TECNOLOGIA E SISTEMAS DE INFORMAÇÃO Veja as fotos da evento: http://www.fea.usp.br/galeria.php?i=202 A FEAUSP recebeu entre os dias 28 e 30 de maio o 11º Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação (CONTECSI), evento multidisciplinar, voltado para os gestores da área. O Congresso objetiva atualizar a comunidade acadêmica e profissional sobre as pesquisas, inovações, regulamentações e outras informações que impactam seus campos de interesse. Previamente, foi realizado o Simulab, um laboratório de gestão em que os grupos participam de simulações que medem o impacto de decisões numa empresa. Ao longo dos três dias de evento foi possível acompanhar palestrantes internacionais, pesquisas de mestrado, doutorado e pós-doutorado em andamento, workshops, dentre outras atividades O programa completo e mais informações sobre o evento podem ser acessadas em http://www.tecsi.fea.usp.br/eventos/contecsi/. É possível também acessar conteúdos de todas as edições do CONTECSI a partir de seu Observatório (http://www.tecsi.fea.usp.br/?q=observatorio), criado no âmbito do convêncio entra a TECSI da FEAUSP e Centro de Estudos em Tecnologia, Artes e Comunicação da Universidade do Porto. 30/05/2014

JISTEM, Brazil Vol. 11, No.2, May/Aug 2014, pp. 461-510

www.jistem.fea.usp.br

JISTEM Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação Vol.11, No.2, Mai/Aug, 2014, pp. 511 ISSN online: 1807-1775

Congresso / Conference

12th CONTECSI International Conference on Information Systems and Technology Management May 20-22, 2015 USP/São Paulo/SP FEA USP São Paulo, Brazil The 12th International Conference on Technology and Information Systems Management CONTECSI is an event focusing Technology and Information Systems Management under a multidisciplinary view. CONTECSI aims at putting together academics and professionals involved in IT and Systems management for a state-of-the-art discussion. International researchers are expected to contribute for the integration between the academic and the professional communities. The Conference welcomes papers submission for presentation and panel discussions. Major topics on interest include, but are not limited to: Information Society, Open Systems, Systems Interfacing and Integration, Wireless Computing, Entrepreneurship in IT and IS, Accounting Information Systems, E-Commerce / E-Business, Software Engineering, ERP Systems, Financial Management in Information Systems, IT Strategic Management. All papers will be subject to a blind review process and full papers will be published (CD) in the Conference Proceedings.

More information: http://www.tecsi.fea.usp.br/eventos/contecsi Coordination: Prof. Edson Luiz Riccio. PhD – FEA USP and TECSI Contact: contecsi@usp.br

Submission Papers Deadline: December 1st, 2014

12º CONTECSI Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação 20 a 22 de Maio de 2015 USP/São Paulo/SP FEA USP São Paulo, Brasil O 12º Congresso Internacional de Gestão da Tecnologia e Sistemas de Informação CONTECSI visa reunir acadêmicos e profissionais envolvidos com a temática de gestão para discussão do Estado-da-arte deste campo. Atualmente este campo encontrase disperso em áreas específicas, carecendo de uma visão holística e integrada do assunto. O CONTECSI contará com a presença de palestrantes de renome, bem como estará aberto para a recepção de trabalhos para serem apresentados em sessões paralelas e painéis. Assim como compareceram nos anos anteriores, são esperados personalidades, professores e pesquisadores do Brasil e do exterior, principalmente de Universidades da França, Inglaterra, Espanha, México, Portugal, Chile, Argentina, Colômbia, Uruguai, Venezuela entre outras. Os foco de interesse deste congresso inclui todas as abordagens referentes à gestão de Tecnologia e dos Sistemas de Informação nas instituições publicas e privadas e na sociedade em geral.

Mais informações no site: http://www.tecsi.fea.usp.br/eventos/contecsi Coordenação: Prof. Dr. Edson Luiz Riccio – FEA USP e TECSI Contato: contecsi@usp.br Data final para envio de trabalhos: 01/12/2014

JISTEM Journal of Information Systems and Technology Management Revista de Gestão da Tecnologia e Sistemas de Informação ISSN online: 1807–1775

Every four months/Quadrimestral

1) Paper Submission Guidelines Register at "Online Submissions" and submit your paper accordingly to JISTEM guidelines at www.jistem.fea.usp.br a) Manuscript style Articles must be submitted in English, Spanish, Portuguese or French in MS-Word format. Authors must translate the final version of the article to English. Fill the submission form with: title of the article, author's full name, affiliation, full address, telephone, email, fax and brief curriculum vitae. Limit of 3 co-authors per article. First page must present: title of the article, abstract in the original language of the article of about 100 words, title, area and 5 keywords (if accepted an abstract in English and keywords will be required), Articles must be limited to 30 pages in double-space, Arial or times new roman, 12 points; Authors must include figures and graphics in high-resolution 300 dpi (jpg or gif). They must be numbered (Arabic) and with the complete title. References to each table or figure have to be made in the text. Authors must submit the questionnaires and research results to the editor and review purposes. Acknowledgments to institutions regarding financial support can be included only in the final accepted version. We do not accept articles published elsewhere, except Conference proceedings. b) Structure Style Articles should clearly present the Abstract, Introduction, Objectives, Justification, Question, literature review, research method, results, conclusion, recommendation and limitation, plus references; References are to follow the American Psychological Association (APA) guidelines. More detailed explanations and examples of these guidelines can be found at the following locations: http://www.apastyle.org/faqs.html or Publication Manual of the American Psychological Association (6th ed., 2010) American Psychological Association (APA).A list of reference must be presented in alphabetical order. A glossary can be included in the end of the article if needed. 2) Book Review Book review should be sent by Prof. Edson Luiz Riccio at jistem@usp.br

JISTEM, Brazil Vol. 11, No.2, May/Aug. 2014, pp. 512-513

www.jistem.fea.usp.br

Contributions / Submissão de Artigos

513

1) Instruções para submissão de artigo a) Quanto à Formatação Os artigos submetidos para publicação, em inglês, espanhol, português ou francês, devem ser enviados em formato MS-Word. Após aceito, os autores devem traduzir o artigo para o idioma inglês. Incluir no sistema de submissão online: título, subtítulo (se houver), tema, nome, instituição, departamento, endereço, telefone, fax e e-mail do autor e co-autores (máximo de 3 co-autores) e breve curriculum que indique sua formação, instituição/empresa a que pertence e sua área atual de trabalho.; Na primeira página do artigo deve constar: título, subtítulo (se houver), tema e resumo na língua original do artigo, com 100 palavras aproximadamente e 5 (cinco) palavras-chaves. Se o artigo for aceito para publicação será solicitado o envio do título, abstract e palavras-chave em inglês; Os artigos deverão ter no máximo 30 páginas em espaço duplo, fonte arial ou times new roman, tamanho 12; As figuras e gráficos devem estar em alta qualidade com resolução de 300 dpi (figuras) e extensão jpg e/ou gif no artigo. Cada ilustração deve conter numeração e legenda. Deve ser feita referência à figura ou tabela no corpo do texto. Questionários e resultados da pesquisa devem ser enviados para a avaliação do Editor e pareceristas. Agradecimentos a órgãos de financiamento da pesquisa devem ser incluídos apenas na versão final do artigo, após o aceite. A JISTEM só aceita artigos inéditos. b) Quanto à Estrutura Os artigos enviados devem conter em seus tópicos os seguintes itens: Resumo, Introdução, Objetivos, Justificativa, Problema/Questão, Revisão da Literatura, Metodologia, Resultados, Conclusão, Recomendações, Limitações e Referência Bibliográfica; As citações e referências devem seguir o estilo da APA (http://www.apastyle.org/l) As referências deverão ser apresentadas no corpo do texto, incluindo o sobrenome do autor, a data de publicação e o número de página (se for o caso), conforme normas da APA. Referências bibliográficas completas do(s) autor (es) citados deverão ser apresentadas em ordem alfabética, no final do texto, de acordo com as normas da APA. Para maiores informações: American Psychological Association (APA). (2001). Publication Manual of the American Psychological Association (5th ed.). Washington, DC Poderá ser incluído um glossário ao final do artigo, caso o autor julgue necessário; 2) Sugestões de livros para Resenha Resenhas devem ser enviadas para o Prof. Edson Luiz Riccio pelo e-mail: jistem@usp.br JISTEM, Brazil Vol. 11, No.2, May/Aug. 2014 pp. 512-513