an evaluation of one pillar of the idb's knowledge and learning strategy: training activities for...

Corporate Evaluation

Evaluation of One Pillar of the IDB´s Knowledge and Learning Strategy:

Inter-Ame rican Development Bank February 2012

Classification: Public Document

This work is distributed under a Creative Commons license (CC BY-NC-ND 3.0). You are free to copy, distribute and transmit this work to third-parties, under the following conditions: Attribution - You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).

Non-Commercial - You may not use this work for commercial purposes.

No Derivative Works - You may not alter, transform, or build upon this work.

Waiver - Any of the above conditions can be waived if you get permission from the copyright holder.

Š Inter-American Development Bank Office of Evaluation and Oversight 1350 New York Avenue, N.W. Washington, D.C. 20577 www.iadb.org/evaluation

TABLE OF CONTENTS ABBREVIATIONS EXECUTIVE SUMMARY MANAGEMENT ACTION RECORD I.

INTRODUCTION ............................................................................................................. 1

II.

TYPES OF TRAINING ACTIVITIES FOR OPERATIONAL STAFF .......................................... 4

III. FINDINGS ...................................................................................................................... 7 A.

Selection ............................................................................................................ 7 1.

2.

B.

Implementation ................................................................................................ 11 1.

2.

3.

C.

Are KNL’s training activities strategically chosen?................................. 7 Are KNL’s training activities demand-driven? ........................................ 8 Are procedures for delivering training effective? .................................. 11 Is information on KNL’s training activities adequate? .......................... 12 Are quality control mechanisms for KNL’s training activities adequate? ................................................................................................ 14

Quality ............................................................................................................. 15 1.

2.

Are KNL’s training activities of high quality? ....................................... 15 Are KNL’s training activities useful? .................................................... 16

IV. SUMMARY AND RECOMMENDATIONS ......................................................................... 19 BOX 1. FELIPE HERRERA LIBRARY

This evaluation was prepared by Guilherme Sedlacek and Viviana Velez Grajales, under the direction of Cheryl W. Gray. Assistance was provided by Andrea Florimon, and comments and inputs were provided by Michelle Fryer, Virginia Poggio, Silvia Raw, and Li Tang. We would like to thank Graciela Schamis and her KNL team for helpful comments on an earlier draft.

ABBREVIATIONS COF

Country Offices

FHL

Felipe Herrera Library

HQ

Headquarters

INDES

Inter-American Institute for Economic and Social Development

K&L

Knowledge and Learning

KCP

Knowledge Country Product

KNL

Knowledge and Learning Sector

KNM

Knowledge Management Division

LACEA

Latin American and Caribbean Economic Association

RES

Research Department

SPD

Office of Strategic Planning & Development Effectiveness

TC

Technical Cooperation

VPC

Vice Presidency for Countries

VPP

Vice Presidency for Private Sector & Non-Sovereign Guaranteed Operations

VPS

Vice Presidency for Sectors & Knowledge

EXECUTIVE SUMMARY In 2007, as part of the Bank’s Realignment and with the objective of strengthening the use of knowledge, the newly-created Vice-Presidency for Sectors and Knowledge (VPS) was given Bank-wide responsibility for knowledge and learning, including developing a demand-driven learning program to build staff skills in support of business needs. In 2008 the Board approved the “Knowledge and Learning Strategy (2008–2010),” which establishes the Bank’s approach to knowledge and learning (K&L) both internally and externally. The Knowledge and Learning Department (KNL) in VPS was set up to oversee knowledge management and internal and external training. KNL has three units: the Knowledge Management Division (KNM), the Inter-American Institute for Economic and Social Development (INDES), and the Felipe Herrera Library (FHL). The K&L Strategy establishes the Department’s objectives and proposes the functions for each one of its units.

The main objective of this evaluation is to identify strengths and weaknesses in the implementation of the Bank’s training program for technical staff engaged in providing operational services to clients. It seeks to investigate whether operational staff are satisfied with the training activities organized by KNL and to what extent KNL’s products and services have been adequately provided and are considered useful. The evaluation does not address other dimensions of the K&L Strategy, including knowledge management or external training, though it does briefly discuss issues pertaining to the Library.

The evaluation finds that KNL has made significant progress since 2008 in developing and consolidating a system for organizing, coordinating, and managing the Bank’s training program. The new system improves the efficiency of the Bank’s training program by addressing the fragmentation of training initiatives and funding sources previously prevailing in the Bank. It seeks to provide transparent and verifiable mechanisms for training budget allocations across and within the Bank’s organizational Units.

KNL has sought to move to a demand-driven system to provide staff training. A member of a Unit’s staff is appointed as the Training Coordinator who assists each Manager in preparing his or her Unit’s Knowledge and Learning Plan. Training Coordinators are responsible for ensuring that training needs are adequately reflected and budgeted in the Plan and activities are adequately executed. To facilitate the process of defining the K&L Plan, KNL assigns a Liaison to help the Unit. He/she provides general information about training, such as which activities are eligible for financing and which ones are not, and helps the Unit ensure that the plan is prepared on time. Weaknesses remain in consultation processes with individual staff about training needs and in mechanisms to link staff training and career development.

KNL has developed a system to disseminate and store information on training activities, which is a key ingredient for a successful and systematized training program. To date the system is not fully utilized, however, as staff use it mainly for registration purposes.

Quality control mechanisms for training events have been put in place and are useful in

-iiassessing the immediate reaction and perceptions of training participants. There is limited evidence of them being used proactively to improve the quality of training and learning or to measure gains in staff competencies.

Operational staff are satisfied with KNL’s logistical and facilitation support for training events. They would like KNL to provide more help in identifying target audiences and in defining learning methodologies and adapting presentation techniques and learning materials to specific audiences. With respect to specific training courses, operational staff find technical courses useful for their work, though they would like more specialized offerings. Retreats are generally considered to be useful in strengthening team work abilities and opening business opportunities, even if they do not necessarily improve staff competencies. Staff agree that new employees and staff transferred to country offices do not receive adequate induction training.

To continue to improve the effectiveness of training for operational staff, this evaluation recommends that IDB Management (in most cases KNL in coordination and collaboration with other Bank Units): a.

Continue to upgrade KNL’s online system to ensure that course descriptions and course feedback are complete and easily accessible (integrating eventually with Optima as appropriate);

b.

Reinforce the requirement that managers and staff in both Headquarters and Field Offices discuss and agree on individual training needs and plans annually;

c.

Establish a clear distinction in terms of objectives and sources of funding between technical training activities and activities (such as Retreats) focused primarily on team-building, to avoid crowding out the former by the latter;

d.

Explore ways to expand training options to meet specialized professional needs, including focused technical courses and costeffective external training;

e.

Strengthen the content of new employee orientation (perhaps through shorter and more frequent offerings) and induction for staff transferred to field offices;

f.

Continue to explore new ways to assess and ensure the quality of training events, including advising trainers on effective communication techniques and following up with staff to apply the techniques acquired during training.

-iiiMANAGEMENT ACTION RECORD An Evaluation of One Pillar of IDB’s Knowledge and Learning Strategy: Training Activities for IDB Operational Staff Recommendations a.

Continue to upgrade KNL’s online system to ensure that course descriptions and course feedback are complete and easily accessible (integrating eventually with Optima as appropriate).

b. Reinforce the requirement that managers and staff in both Headquarters and Field Offices discuss and agree on individual training needs and plans annually. c.

Establish a clear distinction in terms of objectives and sources of funding between technical training activities and activities (such as Retreats) focused primarily on team-building, to avoid crowding out the former by the latter.

d. Explore ways to expand training options to meet specialized professional needs, including focused technical courses and costeffective external training. e.

Strengthen the content of new employee orientation (perhaps through shorter and more frequent offerings) and induction for staff transferred to field offices.

f.

Continue to explore new ways to assess and ensure the quality of training events, including advising trainers on effective communication techniques and following up with staff to apply the techniques acquired during training.

Management Response

I. INTRODUCTION 1.1

Enhancing the competencies of Bank staff through learning and knowledge sharing has been recognized by the Bank as key to better address the needs of its clients and to deepen its development impact. In 2005, as a result of the review of the implementation of the Bank’s New Lending Framework, Management recommended strengthening the Bank’s analytical capacity. In 2007, as part of the Bank’s realignment and with the objective of strengthening the use of knowledge, the newly-created Vice-Presidency for Sectors and Knowledge (VPS) was given the responsibility to “develop a demand-driven learning program for staff skills in support of business needs” (document GA-232, Annex VIII).

1.2

The Knowledge and Learning Sector (KNL) is the Department of VPS responsible for disseminating knowledge and for internal and external training (document GA-232). KNL has three units: the Knowledge Management Division (KNM), the Inter-American Institute for Economic and Social Development (INDES), and the Felipe Herrera Library (FHL). In 2008 the Board approved the “Knowledge and Learning [K&L] Strategy (2008–2010)” (document GN-2479), which establishes the Department’s objectives and proposes the functions for each one of its units. These functions include the identification of organizational Units’ training needs, the design and implementation of the institutional training agenda, and the management of the Bank’s training and learning budget.

1.3

The main objective of the evaluation is to identify strengths and weaknesses in the implementation of the Bank’s training program for technical staff engaged in providing operational services to clients. This evaluation focuses on the implementation, within one important client group, of one of the five strategic objectives laid out in the 2008 K&L strategy: “Continuously update and strengthen the technical, central and supervisory knowledge and competencies of the Bank’s managerial, professional, and administrative staff.” The evaluation seeks to investigate whether operational staff are satisfied with the training activities organized by KNL, often with partner units, 1 and to what extent these products and services have been adequately provided and are considered useful to improve staff competencies and work performance.

1.4

The evaluation does not explicitly cover management training or training provided to administrative or non-operational staff, though some of the findings may be relevant to these programs as well. It also does not cover KNL’s broader knowledge management programs or external training for clients and only briefly touches on library services, although all of these dimensions of K&L are to some extent mutually reinforcing and are important in developing IDB as a knowledgebased institution.

1.5

Core criteria used for this evaluation include the following:

1

The Strategy expects KNL to integrate horizontally with partner units, combining its expertise in the promotion of learning with the contributions of subject matter specialists in other units.

-2-

1.6

•

K&L work should be relevant to the operational needs of the institution and its clients, and should specify its anticipated contribution at the outset.

•

K&L work should be programmed to establish priorities, avoid duplication of effort, and take maximum advantage of synergies within the institution.

•

K&L work should be budgeted to know in advance how much of the institutions scarce resources are being committed to a given effort, and tracked so as to determine compliance with programming.

•

K&L work should have quality control mechanisms to ensure that products and services meet specific quality standards.

•

K&L work should be disseminated as a way of diffusing the benefits of training throughout the relevant community.

•

K&L work should be stored in a manner that allows future users to find and make use of the learning and knowledge.

•

K&L work should be of good quality to ensure that the institution is receiving value for the money invested.

•

K&L work should be useful in the development of operational programs.

Specific evaluation questions are as follows: Selection • •

Are operational training activities strategically chosen?

Are operational training activities demand-driven?

Production processes • • •

Are procedures for delivering training effective?

Is information on training activities adequate?

Are quality control mechanisms for training activities adequate?

Quality • •

Are operational training activities of high quality?

Are operational training activities useful?

Unfortunately it is not possible to compare training efficiency and effectiveness before and after 2008 to analyze how KNL efforts have improved the Bank’s training program as a result of the implementation of the K&L Strategy, because data on training prior to 2008 are not available.

1.7

The sources of information for this evaluation include the following: •

KNL’s website. The website explains the main functions of the Units and presents a brief description of the products and services they provide. It contains links to documents relevant to this evaluation, including the approved Knowledge

-3and Learning Strategy, 2008-2010 program and budget reports, and guidelines to prepare the Knowledge and Learning Plans (K&L Plans) of the Bank’s organizational Units. The website also provides access to the online KNL system, where employees register to events organized by KNL and where information associated with these events, such as agendas and presentations, can be found.

KNL presents in their website reports on the results of the evaluation of their training activities, consisting of surveys filled out by participants at the end of the event.

•

KNL’s database and the Bank’s employee database. The KNL database contains information on the name, objectives, cost, duration, and OPUS codes for each event organized by KNL. This information can be analyzed in various ways depending on how events are grouped. For instance, events can be classified by type (some addressing specific training needs of a Unit, others addressing Bank wide training needs) or by purpose (some seeking to enhance technical abilities, other seeking to improve language skills). That database also provides information about participants, such as their organizational Unit, their location (HQ or Country Office), and their title. Similar information can be obtained for non-participants from the Bank’s employee database.

•

Interviews. A broadly representative group of 34 staff from the three operational Vice Presidencies (VPC, VPS, and VPP) and SPD were interviewed to gather information on opinions about KNL’s performance regarding the provision of training. Eighty-percent work in HQ and 20% in Country Offices. Interviewees were selected randomly from groups of Specialists and Senior Specialists (17), Young Professionals (3), and Training Coordinators (7) 2 . In addition, 7 staff were selected purposively to include representation from RES, SPD, and more senior and junior grade levels in operations. Interviewees had interacted with KNL in different ways. Most of them had participated in at least one event. Others had also worked together with KNL to organize an event. A few others had been in charge of preparing training plans for their Unit. Depending on the kind of relationship they had with the sector, interviewees were asked questions about the quality of events, whether they learned something useful for their work, and how often they used the services of the library.

•

Survey. A survey was sent electronically to the 292 Specialists and Senior Specialists who participated in training activities, 57% in Country Offices and 43% in HQ. 3 A total of 113 individuals responded to the survey, which represents

2

Groups of staff are classified according to their position title as reported in the Bank’s employee’s database.

Specialists and Senior Specialists, Associates, and Young Professionals are considered mid-level staff. Lead Specialists, Principal Specialists, Division Chiefs, Unit Chiefs, Advisors, and Representatives are considered upper-level staff. Training Coordinators hold diverse position titles and were interviewed because of their close interaction with KNL.

3

Specialists and Senior Specialists are the core group of operational staff responsible for much of the technical work on Bank projects. They account for about half of the professional staff in the three operational Vice Presidencies (VPC, VPS, and VPP). The 292 employees to whom the survey was sent account for around 90% of the Bank’s Specialists and Senior Specialists. The other 10% did not participate in a training event.

-438% of those contacted. From those who responded, 67% work in COF and 33% in HQ. The distribution of respondents according to the training event they were asked about (although not necessarily the only one in which they participated) is shown in Table 1. More than half of respondents (55%) participated in a Sector & Economic Analysis activity, the most common training activity of this group of operational staff.

Table 1.

Distribution of survey respondents by training event they participated in Training activity classification

Percentage

Sector & Economic Analysis (BK-C1014)

55%

Country Team Workshop (BK-C1006)

18%

Core Competencies (BK-C1011)

15%

Systems & Software (BK-C1015)

5%

Operational Policies & Procedures (BK-C1013)

4%

Corporate Management (BK-C1012)

3%

Total

100%

Source: KNL's database

II. TYPES OF TRAINING ACTIVITIES FOR OPERATIONAL STAFF 2.1

According to the K&L Strategy, KNL’s goal is “to improve conditions for the acquisition, creation, dissemination, sharing and use of knowledge needed for the effectiveness of Bank operations, in accordance with the region’s needs and development challenges.” KNL is responsible for designing, developing and managing the Bank’s training program together with organizational Units, promoting the utilization of knowledge, disseminating lessons learned, establishing knowledge partnerships with academic and professional networks, and supervising the FHL.

2.2

Concerning training, the role of KNL is to support Units in the design, execution, and evaluation of their training agenda. During the design process, assistance is provided to Units to assess training needs, identify training objectives, establish techniques for delivering training, and allocate resources. There are two types of training activities, those planned by Units with the support of KNL and those planned exclusively by KNL that seek to address cross-sectoral training needs.

Although financial resources are allocated to Units for the first type of activities, KNL is in charge of managing the budgets. During the execution phase of the training program, KNL develops training methods, prepares training materials, and helps with the logistics of events. At Units’ request, KNL provides the services of a moderator to facilitate the communication between trainers and participants. Finally, KNL monitors and evaluates the effectiveness of training.

2.3

As mentioned above, training activities are divided into two groups depending on the target population. Those planned by the Units in their K&L Plans address

-5training needs of Units’ staff and counterparts (Needs, Contributions and Outreach KCPs), 4 and those defined, organized and financed exclusively by KNL (Bank-Wide activities) support the learning of cross-sectoral competencies and corporate processes. Within these two groups, training activities can be further classified according to their purpose: • Core Competencies (BK-C1011) training activities enhance staff skills on language, communication, and negotiation and provide an institutional overview of the Bank to new employees.

• Corporate Management (BK-C1012) training activities strengthen staff ability to apply new technologies and perform new responsibilities, and to provide staff with practical knowledge on the application of Bank policies and regulations.

• Operational Policies & Procedures (BK-C1013) training activities deepen technical capacity and skills of staff that work mainly in the operational area such as project team leaders.

• Sector & Economic Analysis (BK-C1014) training activities improve staff skills, knowledge and technical competencies.

• Systems & Software (BK-C1015) training activities strengthen staff ability to use information systems and computer applications and update staff knowledge on policies and procedures.

• Country Team Workshops (BK-C1006) seek to enhance the organizational effectiveness of Bank Units by clarifying roles and responsibilities of staff and addressing issues of team work and communication.

2.4

4

Descriptive statistics are presented below for 2010 training activities for operational Specialists and Senior Specialists. 5 The information is presented in two parts: training activities planned by Units (Needs and Contributions) and those planned by KNL (Bank-Wide activities). 6

Needs seek to address the current and future knowledge and learning demands of the organizational Unit and its staff. They include courses for strengthening core competencies and technical competencies.

Contributions are activities planned by a Unit that benefit other organizational Units. For instance, lectures on departmental processes usually are organized by the Vice Presidencies but are targeted to staff of divisions and departments. Outreach Knowledge Country Products (KCPs) are products that address the learning needs of the Bank’s counterparts, and they are not included in this evaluation.

5

The database on training activities was revised and improved by KNL in 2010.

6

The classification of activities by Needs, Contributions, Outreach KCP and Bank-Wide products needs to be revised in the KNL’s database. According to KNL’s Business Plan, only Core Competencies (BK-C1011), Corporate Management (BK-C1012), Operational Policies & Procedures (BK-C1013), Sector & Economic Analysis (BK-C1014), Systems & Software (BK-C1015), Country Team Workshop (BK-C1006), and Communities of Practice (BK-C1005) can be identified as Needs by the Units. However, other types of activities are reported as Needs in the database such as Documentation & Dissemination of Lessons Learned (BK-C1004) and Knowledge Portals (BK-C1008).

-62.5

Operational Specialists and Senior Specialists participated in 156 training activities planned and carried out by Units during 2010 with the support of KNL. 7 The distribution of these activities classified according to their purpose is presented in Table 2. More than half correspond to Sector & Economic Analysis activities, which are also the most expensive in terms of average cost per participant. An average of 24 participants registered for each of these activities.

Examples include “STATA,” “Competencias Técnicas Sectoriales – Desarollo Urbano Integrado, Vivienda, Temas Fiscales y Asuntos Sub-Nacionales y de Descentralización”, and “Participation Climate Change Impacts on Water: An International Adaptation Forum.”

2.6

The second and third most common type of Unit-planned activities in which Specialists and Senior Specialists participated are Core Competencies and Corporate Management. Notice that core competencies training present the highest average number of courses (3.9) and participants (87) per activity.

Examples of these activities include “Participation in a course to obtain a Project Management Certificate,” “Presentaciones de Especialistas en el tema de Competencia Workshop,” and “Taller sobre habilidades comunicacionales y manejo del conflicto”.

Table 2.

Descriptive statistics of 2010 activities planned by Units where at least one Specialist or Senior Specialist participated

Training activity classification

Average Average number of number of Number of events/courses participants activities per activity per activity

Average Average cost per per capita cost * activity

Sector & Economic Analysis (BK-C1014)

87 (56%)

2.7

24

11,929

1,248

Core Competencies (BK-C1011)

21 (13%)

3.9

87

4,970

805

Corporate Management (BK-C1012)

21 (13%)

3.5

46

2,836

485

Country Team Workshop (BK-C1006)

15 (10%)

1.1

25

7,098

300

6 (4%)

2.1

40

48,968

676

6 (4%)

2.6

19

523

46

156 (100%)

2.8

36

10,290

926

Operational Policies & Procedures (BK-C1013) Systems & Software (BK-C1015) Total Source: KNL's database

2.7

7 8

According to KNL´s database, 18 Bank-Wide activities were carried out during 2010 in which Specialists and Senior Specialists participated. 8 The distribution of these training activities classified according to their purpose is presented in Table 3. More than 60% fall into the group of Core Competencies. Training is provided through an average of 12.8 courses per activity, with an average of 96 participants per activity. Examples include “Spanish, French, Portuguese and English Group Classes,” “New Employee Orientation” and “Delivering Presentations.”

A total of 552 staff training activities planned by Units (Needs and Contributions) took place during 2010.

A total of 28 Bank-Wide training activities targeted to staff were carried out by KNL during 2010.

-72.8

The least expensive Bank-Wide activities (an average of US$76 per participant) with the largest number of registrations (an average of 492 per activity) are classified as System & Software. The “Standard Bank Computer Applications Program 2010” consists of information sessions on the electronic systems of the Bank: OPERA, OPMAS, SISCOR, IDBDOCS, and Unified Communications’ System. The “Computer Applications Program 2010” provides training on Microsoft 2007 Applications.

Table 3.

Descriptive statistics of 2010 Bank-Wide activities where at least one Specialist or Senior Specialist participated

Training activity classification

Average Average number of number of Number of events/courses participants per activity per activity activities

Average cost per activity

Average per capita cost

Core Competencies (BK-C1011)

11 (62%)

12.8

96

46,279

550

Sector & Economic Analysis (BK-C1014)

3 (17%)

1

26

4,903

136

Systems & Software (BK-C1015)

2 (11%)

47.5

492

35,662

76

Country Team Workshop (BK-C1006)

1 (5%)

6

252

291,235

1,155

Operational Policies & Procedures (BK-C1013)

1 (5%)

5

67

84,164

1,256

18 (100%)

13.8

135

53,917

501

Total Source: KNL's database

III. FINDINGS 3.1

Evaluation findings on staff training are described below. Box 1 contains a summary of some additional findings on the Felipe Herrera Library obtained during the course of the evaluation.

A.

Selection 1. Are KNL’s training activities strategically chosen?

3.2

Guidelines are available for the elaboration of the annual Knowledge and Learning Plan for each organizational Unit at the beginning of each year. They contain information about the definition of and amount of resources allocated to Needs, Contributions, and Outreach KCPs by Unit and explain the four stages of the planning process: identification and prioritization of knowledge and learning needs, in-depth analysis of prioritized needs, analysis of alternative responses, and budgeting & programming of activities. They also establish deadlines for deliverables and provide a brief description of responsibilities of the different actors involved in the process.

3.3

KNL pre-identifies Bank-Wide needs before organizational Units prepare their individual plans. Managers and Unit Chiefs make the final decision on how to

-8prioritize training activities for their Units. Given their knowledge about the training needed to improve the effectiveness of their Unit’s work, managers should be in a position to make well informed-decisions. The major complaint heard from interviewees is not the locus of decision-making responsibility but rather the shortage of training funds, which may constrain strategic choices.

3.4

This planning process has significantly increased the strategic focus and coherence of the training program as compared to the fragmented and loosely planned system that prevailed prior to the 2008 K&L strategy.

2. Are KNL’s training activities demand-driven?

3.5

KNL has sought to move to a demand-driven system to provide staff training.

Managers prioritize training needs and decide on which activities to finance. To facilitate the process of defining the K&L Plan, KNL assigns a Liaison to help the Unit. He/she provides general information about training, such as which activities are eligible for financing and which ones are not, and assists the Unit to ensure that the plan is prepared on time. A member of a Unit’s staff is appointed as the Training Coordinator to assist the Manager in preparing the Unit’s Knowledge and Learning Plan. Training Coordinators are responsible for training needs being adequately reflected and budgeted in the Plan and activities being adequately executed.

3.6

KNL’s training activities are demand-driven from the point of view of Managers but not necessarily from the point of view of the Unit’s staff. Interviewees agree that training needs are identified and training proposals are prepared by the Unit.

Those who had an opinion about the participation of the Unit in the definition of objectives, contents and learning methodology of activities acknowledged that it indeed participates. Three-fourths of interviewees report that their Unit participates in the definition of objectives. Two-thirds think the Unit also participates in the definition of content, and one-half believe the Unit participates in the specification of learning methodologies

3.7

Although data on Unit K&L budgets can be found on KNL’s website, communication between Unit Managers and their staff on training plans appears to be limited. Staff do not appear to be well informed about the process for preparing K&L Plans or about budget availability. Only 29% of surveyed Specialists knew their Unit’s K&L Plan in 2010, and only 61% of surveyed Specialists knew who the Coordinator in their Unit is. Of the 74% who talked about training budgets, one-third did not know the amount potentially available for their individual training.

3.8

Furthermore, a high proportion of staff do not appear to be consulted about their training needs, and it is not clear whether there is a formal mechanism for staff to request training. While some interviewees requested training in their Annual Performance Review and others as part of surveys and consultations conducted by the Training Coordinator, many were never consulted. Indeed, about two-thirds of

-9interviewees mentioned they were not asked about the kind of training they needed for the year.

3.9

In addition, staff are uncertain about how the budget for unanticipated training needs is administered. More than one-third of interviewees pointed out that the mechanism to finance future training not included in the K&L Plan is not specified. Staff who asked their supervisors for funds for courses advertised later in the year were sometimes turned down for unexplained reasons.

3.10

Finally, one concern brought up often by interviewees is the weak link between the staff’s career development and the needs identified as part of the Unit’s training plan. IDB’s K&L strategy is focused on institutional business needs rather than on individual career development, and the Career Management Framework is meant to support career development. However, staff do not fully understand this focus, and this is leading to misunderstanding and dissatisfaction.

Two-thirds of interviewees talked about this issue. Less than one-third of them consider themselves as having received training connected with their career development, and the rest report never having a discussion with their supervisors about their career in the Bank. In general, Bank staff consider that their training is not adequately linked to their future work at the Bank.

3.11

The following comments illustrate the frustrations of some staff with career development: •

“I think that the Bank does not do a good job on training. The Managers do not have a good vision of where they want you in your career. In general, there is not a good Human Resources strategy. The Bank does not understand what competencies the staff has. They don’t know what they want me to do in 5, 10, 15 years, so how are they going to train me?”

•

“We want to be a Bank of knowledge. I don’t see that KNL is in line with the Human Resources strategy in getting talent, developing talent and keeping talent.”

•

“There are no career paths for management positions. We need a more structured training, maybe with some flexibility, a template that tells you the courses you should take if you are in a certain area. KNL just gives you a long list of courses. I usually just take the courses that I am told to take.”

-10Box 1. Felipe Herrera Library The Felipe Herrera Library is another part of the KNL system. Though it is not the focus of this evaluation, the evaluation team did gather some feedback from interviewees on their use of Library services. The Library provides information on topics relevant for the Bank’s work to both staff and Bank clients. It identifies, recompiles, organizes and disseminates books, articles, newspapers, databases, working papers, etc. The FHL subscribes to individual databases, individual journals, reference resources, the Elsevier Package, and Wiley/EBSCO. Other services provided include the InfoGuides which are WebPages created by the library staff to provide useful and updated information for specific groups of Specialists, alerts sent by email, inter-loan library service, Rosetta Stone Software, and portals that provide access to working papers, news, statistical yearbooks, encyclopedias, and The Economist.

As technology has changed and the amount of available information in electronic form has increased, libraries have developed more complex ways to store and disseminate knowledge. Now, users do not have to go to a building to have access to information, but they still need help to locate it. The role of librarians is evolving, as they not only have to acquire, organize, and store information, but they also have to teach users how to find it. Also, users demand easier and faster access to what they are looking for. Because of these changes, the FHL is implementing a new service model that requires increased interaction and communication between the library and staff to better understand the demands of users. The FHL assigned Liaisons for each sector who are responsible for being in constant contact with its staff to better understand their information needs and in this way provide them with better service. For instance, for the InfoGuides project, Liaisons make sure that the content of the websites is selected and organized according to staff demands. Also, for the electronic alerts, they review external sources of information and send the most relevant links to Specialists. With the Liaison service model the library seeks to store and disseminate information more effectively. Although not experts of the sector, Liaisons are becoming experts as information officers of the sector.

Concerning the quality of library services, more than two thirds of interviewees and about the same proportion of surveyed Specialists find the inter-loan library service of good quality, though a few would like to have more immediate access to the requested articles. However, staff consider that the FHL’s book collection is outdated and the journal collection is limited. The FHL acknowledges that it needs to update its collection of books and is studying the possibility of buying more licenses for e-books because their demand is increasing faster than the demand for printed books. Although a formal mechanism for the prioritization of subscriptions to electronic information resources has not been put in place yet, some Units report that recently the FHL asked their opinion about which journals to subscribe to in their area of interest. Also, OVE understands that since last year the library has been requesting vendors to provide usage statistics on journals and databases.

An important limitation of the FHL is that a high proportion of Specialists do not use their products and services. Surveyed Specialists were asked to rate their level of satisfaction with a list of 17 services. These include the utilization of books, e-journals, inter-loan library service for articles, InfoGuides, alerts sent by email, The Economist, and working papers portals. For each one of the services at least 40% of respondents have never used the service. For instance, 47% of Specialists have never gotten an e-journal and 48% have never borrowed a book. The services with the lowest level of utilization (70%) are the portals for dictionaries, directories and encyclopedias.

COF Specialists use the services of the library even less frequently than HQ Specialists. In general, more than 50% of Specialists in country offices do not use each one of the 17 library services the survey asked about. For instance, 52-53% of COF Specialists have never borrowed a book or e-journal or even done a search for those products using the catalogue or the e-journals search engine. These percentages are lower for HQ Specialists, between 16 and 21%. The reasons for this problem need to be studied.

-11B.

Implementation 1. Are procedures for delivering training effective?

3.12

About half of interviewed staff participated in the organization of a KNL event.

Administrative processes are more transparent than before, though most organizers would like greater flexibility, especially in managing training budgets.

In general they think that the process could be less complex and time consuming given the kind of events being planned and the size of the budget approved. Some interviewee comments include the following: •

“KNL had good experts and materials, but they spent a lot of time on how to pay the experts, how to divide the lunch time, etc. I thought they were not really helping because I had to select the content but had discussions for a few dollars with KNL for everything.”

•

“Fui coordinador de capacitación en la representación. El plan de capacitación que me tocó hacer fue un delirio en cómo presupuestar. En un afán de perfección y aprovechar economías de escala, lo que KNL proponía hacer era demasiado sofisticado para lo que se tenía que planificar. A la mitad del proceso KNL cortó el proceso y aceptaron los planes como estaban. El siguiente año hubo un cambio fuerte.”

3.13

Most interviewees consider that KNL does a good job in organizing training events by providing good support in registration, pre- and post-event communications, travel arrangements, material distribution, audio visual equipment installation, location and room arrangements, and event evaluation. A few staff complained about KNL hiring consultants to help with the logistics of their events who did not know adequately the business of the Bank, which caused organizational problems in managing the event. Surveyed Specialists and Senior Specialists also believe that KNL provides good service when organizing events.

Around 69% of respondents think that the logistics of the event were of good quality, 9% rated it as low quality, and the rest did not have an opinion.

3.14

Staff would like stronger engagement from KNL in defining learning methodologies. About 60% of interviewees had an opinion about this, and 80% of them wanted KNL to provide better service in this area. Some of them believe that KNL is willing to introduce new learning methodologies but does not yet have the expertise to do so. OVE heard both positive and negative views: •

“KNL gave us advice on how to put together the presentations. I think they are taking the role of organizers and also methodology designers. I organized a module for the retreat. It is important to have activities that help improve the interaction with the audience. I asked KNL about how to do it, and got very good insights. It is the customers who should ask for that help.”

-12•

3.15

“In a technical course I organized, KNL was just doing the administrative part instead of participating as a knowledge expert. They discussed whether the room should be set up in some specific way but I didn’t get any new information, I didn’t get help with the presentations format, or any other more important issues.”

Around half of interviewees wanted KNL to know target audiences for training events better. The rest did not bring up this point. Characterizing training participants in terms of their level of knowledge on the topic and their training needs helps in setting the objectives of the event and in selecting an adequate teaching and learning methodology for conducting the training. Moreover, knowing the audience facilitates the interaction between the trainer and the trainees. As noted by one interviewee: •

“In terms of how to communicate more effectively it depends on the audience, and sometimes KNL is aware of the audience they are targeting and others not.”

2. Is information on KNL’s training activities adequate?

3.16

As a key necessary element to manage the Bank’s training program, KNL designed a system in 2008 to advertise training online, where staff can browse the list of events financed and/or organized by KNL, register for them, and download information such as the agenda of the event, printed materials, and, for those that already took place, a summary of the results of the evaluation survey with participants’ opinions on the satisfaction, relevance and utility of the activity.

Staff are authorized to access different kinds of information depending on their roles in the training process. Managers and Coordinators have access to more data than regular staff, such as the allocated amounts of training resources for the Unit and aggregate allocations across Units.

3.17

In general, staff consider that KNL has done a good job in concentrating information on training events in the KNL system. Half of interviewees think that advertisement and description of events is adequate, while one-fourth think they are not adequate and one-fourth have no opinion on this issue. Some suggest that KNL should not only list the events available but also guide staff in deciding which courses to take. Regarding the advertisement of external events, only 40% of interviewees discussed this point, but all of them agreed that it is not adequate.

Usually staff do their own research on the availability of external events by searching websites of other multilaterals, universities and training institutions.

3.18

Most staff use the KNL system only for registering for events and not necessarily to look for them. Most of them attend the events they are invited to. From the results of the survey, 49% report having received an invitation from their Unit to participate in a specific event. Only 2% report having found the event while browsing the KNL system. Another 12% received an invitation from KNL, 16% were invited by other Unit or someone else in the Bank, and 8% of them looked

-13for the event in other institutions. The rest were either organizers or were invited by another institution.

3.19

3.20

Staff frequently expressed the view that the availability of reference materials and manuals for the use of the Bank systems critical for their work is limited. More than 80% of those who had an opinion on this aspect agreed that having adequate manuals or the availability of a help desk would have facilitated their learning of procedures and processes of the Bank. In particular, the information provided in the induction courses could be easily recorded and made available to all staff.

Interviewee’s comments include: •

“For learning OPUS, IDBDOCS, etc. handouts with detailed instructions and sometimes a video would be useful. If you have questions, you could contact someone.”

•

“It would be easy to prepare manuals for operational processes. There are manuals in the webpage but they are for people who already know something.

I didn’t know the language. It wouldn’t be expensive or hard to make a manual explaining the concepts and the processes of operations.”

•

“In my Unit, new employees get a binder with the policies and procedures, the travel requirements, information on the staff’s responsibilities. It is a great way of put you up to speed. Then, the chiefs talk to you. I learned about project management through documents that other staff put together. I also learned about links to KNL, and in KNL I learned of other links.”

•

“Maybe a manual would help for minor questions. But for managing operations I would prefer to talk to someone. I think it is not a KNL issue but a Bank issue. The culture within the Bank is very different in country offices compared to HQ. In principle, a help desk would be a good idea but I don’t know how efficient that would be. If it is a long process maybe you prefer to ask your colleague in the next door. That is faster. The best would be to have practical examples.”

While IDB staff recognize that the Bank is trying to change its way of doing business and become a more decentralized institution, staff also have the perception that many of the training events are targeted to HQ Staff. Interviewees mentioned that in the Bank there is the idea that knowledge is generated in HQ and that COFs are only an operational center. Interviewed COF staff expressed the view that Bank training is still often designed as if all substantive operational decisions were made in HQ. For instance, it was noted that the content of the integrated project management course in COF was very basic, as if the audience did not know the concept. In addition, some think that not enough effort is put into making events accessible to field staff. One of the reasons cited is that attendance at an event must be authorized by their division’s Manager in HQ. Another is the cost in time and budget associated with attending events at HQ or in other countries.

-143. Are quality control mechanisms for KNL’s training activities adequate?

3.21

KNL has developed a system to gather information on participants’ opinions about the events through post-event surveys. All events that are at least four hours long are evaluated at “Level 1,” which measures satisfaction with course content and delivery. Longer courses (some events of 24 hours or more and all events of 40 hours or more) also receive “Level 2” and “Level 3” evaluations measuring learning and application of knowledge and skills, respectively. KNL has prepared standard templates for Level 1 evaluations and has a team that supports staff from KNL and other organizational units in the preparation and deployment of more tailored evaluation surveys for longer courses.

3.22

Though quality control mechanisms for training events have thus been put in place, there is limited evidence that the data are being used to improve the quality of training. Organizers agreed that the surveys were a useful instrument to assess the immediate reaction and perceptions of the participants, but the majority (80%) of them mentioned that evaluation surveys may not be effective in assessing the utility and relevance of courses. The view of one interviewee was as follows: •

“The surveys are a very loose measurement of success, not valid, reliable or precise. Success is ill or very loosely defined. Captures the subjective impression of participants - not how much learning was achieved. They do not have a baseline and should have one. The questions are mostly pro-forma.”

3.23

Another interviewee suggested doing a follow-up survey a few months after the event took place to better assess whether the participant actually acquired the expected competencies. There is a need for continued experimentation with methods to assess impacts of training events on participant’s competencies and learning.

3.24

Tabulations of the responses to the surveys are intended to be distributed to organizers. But only 50% of interviewed organizers reported receiving the survey feedback. It should be noted, however, that those who did not receive the survey tabulations acknowledge not asking for the survey ratings. For the surveys to be a good quality control mechanism, both KNL and Units need to study the participants responses to be able to improve the quality of training by revising the objectives, learning methodology or content of the event.

3.25

The experience of most organizers interviewed (80%) is that KNL does not provide feedback on their performance. Regarding learning methodology, most trainers do not receive after-event feedback on their performance vis-à-vis presentation skills, the format of the presentations, adequacy of the presentation to the audience, or adequacy of trainer-trainee interaction. All interviewees noted that KNL did not ask for feedback from the Unit on how effectively it performed its task, including asking the Unit’s opinions about the vendors.

-15C.

Quality 1. Are KNL’s training activities of high quality?

3.26

KNL attempts to address this question using end-of-course surveys, among other resources. KNL requires a Level 1 evaluation (which measures satisfaction with course content and delivery) for all training that has a duration of four hours or more. The percentage of evaluable events that are effectively evaluated (compliance rate) stands at 87.7%, somewhat below the 100% target. The sector considers that a feedback response rate from the users of 60% for these evaluations is satisfactory, and in fact they report a response rate of 70.9%. The compliance and response rates could usefully be increased, along with efforts to harvest lessons from past experiences and apply them to future training sessions.

3.27

KNL’s goal is to get satisfaction scores in these end-of-course evaluations of at least 4 out of 5. OVE looked at reports on KNL’s Level 1 evaluations of 172 internal non-language events in 2010. The data are used to calculate a Quality Index (QI), a weighted average of 4 components: facilitation, content, methodology, and materials scores. The average composite Quality Index for the 2010 courses was 4.26, with the “contents” component being the lowest-scoring one, at 4.15. Just under 20% of courses (32 out of 172) had a QI of less than 4.

3.28

Of interviewees who took a technical course, 54% were satisfied with the quality of the content and 46% were not. Most thought that the events were not specific enough to improve their competencies, and around 80% would like to participate in more specialized courses. Examples of training topics that staff mentioned to improve their technical competencies include finance, econometrics, STATA, evaluation methods, sustainability, design of surveys, etc. Comments of interviewees include the following:

3.29

•

“Their courses seem to target very general skills, not as relevant for what I do.

I think I need more specialized courses. The role of KNL for these types of courses should be coordinating, financing. But the content should be chosen by specialists.”

•

“The division should concentrate on having experts in some areas. For training they should focus, there is certain resistance to move from broad training to a more focused one. The level of knowledge has to be different now, the countries are better in what they do. The Bank has to put the funds in knowledge if the Bank wants to go that way. The Bank is not really serious about it.”

The survey results were somewhat better. Surveyed Specialists and Senior Specialists rated the quality of the content of technical events slightly higher than the sample of interviewed staff; 77% of Specialists rated it of high quality, 8% of low quality, and 13% as neutral. This finding is consistent with staff opinion that in general the Units are defining the content of events adequately.

-162. Are KNL’s training activities useful?

3.30

A high proportion of surveyed Specialists and Senior Specialists (80%) find technical events useful for their work. In particular, around 40% of them report what they learned being useful for dialogue and establishing a good relationship with stakeholders or other staff. However, less than 20% applied the acquired knowledge in preparing a study, report, country strategy or sector strategy which, as discussed above, could mean that technical training is not specific enough for improving their technical competencies.

3.31

Staff found the Orientation for New Employees course of little use. This activity is classified as Staff Training on Core Competencies (BK-C1011) and is currently offered every six months. Around 40% of interviewees talked about their experience with the induction course and all of them agreed that it was of little value, either because it was too general or because it was offered to them too late.

Those who took the course a few days after they started working believe that the information provided is too broad and did not gave them tools to really understand how the Bank works. The others affirm that the course was offered to them months and, in some cases, years after they arrived to the Bank, and they decided not to take it. For most of them, their introduction to the systems and procedures of the Bank was done in an informal way (on-the-job) with the help of colleagues. A few of them were given documentation about their Unit’s work.

They think that was useful but not the best way to learn because sometimes it was incomplete.

3.32

The interviews revealed a unanimous view that the Bank does not provide adequate training opportunities to staff being transferred to the Field. While the competencies required to work in the COF often overlap with those required to work at HQ, transfers to COF are associated with changes in staff task responsibilities to now certainly include those related to the implementation of Bank financed projects and TCs. The Bank has not prioritized formal training on new functions and systems for people who are transferred to a COF. All interviewees who were transferred to COF affirm they did not receive training on local systems. Some of their comments are:

3.33

•

“When I was transferred, I received no training courses about the processes and systems of the country office which are very different from those in HQ.”

•

“Before coming to the COF, I should have learned more about procurement.”

•

“I took the induction course half a year later, I think it is a big problem particularly because I was hired in a COF. Before, I was a consultant in HQ but didn’t know about the execution processes that take place in the COF, I needed to know new tools. Nobody explained anything.”

The staff found training events on other Core Competencies courses more useful.

Half of staff took courses on presentation skills, writing skills, and languages. In

-17general, they found the presentation skills and writing skills courses useful. They say that when they prepare presentations in Power Point or write a document, they apply what they learned. In the case of the language courses, there were both positive and negative views: •

“I took all courses on Portuguese in groups of 4 to 6 people. The professors were good. Now, I am working with Brazil, and all conversations are in Portuguese.”

•

“I enjoyed the Spanish classes, the structure is good, other issues are about your classmates, what they put of effort affects the course. If it is a small group, it puts pressure on the teacher to retain people. Usually you have two classes a week, a total of between 30 and 44 hours.”

•

“I tried to take Portuguese, but gave up the 3-hour-a-week course. It was going very slow and I know that because I went on a mission and when I came back they were in the same topic. I think it should be harder, force you to work. If those three hours were useful I wouldn’t mind spending that time.

You need the conversations, the interaction with a counterpart. A different format would be more useful because it is up to the person to really try it. In the firm I was before they gave us online Rosetta Stone and it was very useful.”

•

“I was taking Portuguese, they are group courses and I was traveling a lot.

You get individual courses only if you travel more than 100 days. So, basically, only the people who don’t travel can take that course like the assistants and lawyers, but they don’t need it. Many people take it for leisure.

For the language courses they don’t do a good job. I don’t understand why KNL doesn’t do something like the IMF, assign individual teachers for a few weeks intensively.”

3.34

A significant proportion of staff find the courses on the systems of the Bank such as IDBDOCS not useful. Those are classified as Staff Training on Systems & Software (BK-C1015). From those interviewees who had an opinion (62%), 81% did not like them. Some thought the systems are too complicated. Others noted that trainers did not know the answer to questions of the audience and that the technology used for virtual courses did not work properly. The other 19% thought that the courses were acceptable.

3.35

Staff found team-building activities in Retreats very useful but agreed that Retreats should not necessarily be considered as training. Although there is uncertainty about whether Retreats should be eligible for financing from KNL, several units have organized them with KNL’s support. Around half of interviewees had an opinion about Retreats. The events they referred to as Retreats are classified both as Sector & Economic Analysis (BK-C1014) and Organizational Development Program (BK-C1006) in KNL’s database. In general, staff consider that the combination of technical training activities and

-18team building activities of Retreats have been useful for improving the work environment and opening business opportunities with clients, and that KNL provides good methodology for team building activities. A few pointed out that not all activities are designed to improve staff competencies. Some comments included: •

“KNL taught us how to work on a team, they told us about some steps we have to follow.”

•

“In organizing Retreats KNL is good. The last one I attended was very successful. People were optimistic, cooperative. It is a bit of a management event and also a training event. I am not a fan of these methodologies. KNL had two roles. They were facilitators for opening a space for improving the way our Unit is organized. They also showed us how to build a team, this last part is important for our Unit because it is very big.”

•

“They try to do technical Retreats, but it is not really training because we don’t have the resources. We are targeting our clients; then, we should be bringing some relevance for the region.”

3.36

The main issue identified is that some Units utilize the training budget to fund Retreats and this decision often leaves a very small remaining budget to fund other training events. While most staff agreed that the organization of Retreats is an effective and necessary Bank business activity, they also agreed that financing Retreats from training budgets often restricts the availability of training opportunities for staff.

3.37

Options for individual training are considered limited. Of those interviewees who had an opinion about this issue (65%), 77% think that it is complicated to obtain funding for individual training for conferences such as LACEA, even when giving a presentation. Someone mentioned that KNL approves funds for those events depending on the wording you use to refer to them; they suggest that instead of saying that it is a conference, staff have to say that it is a training event. KNL reports that funding to attend conferences of professional associations was made easier in January, 2010. It appears that not all staff are aware of those changes.

3.38

A significant proportion of surveyed Specialists and Senior Specialists consider that the best way of acquiring the skills they need to better perform their job at the Bank is through training outside the Bank (though it should be noted that this is also likely to be a relatively expensive option). Forty-two percent think the best option is to get trained outside the Bank, 23% prefer on-the-job training, and 22% would like to get trained at the Bank through KNL. The rest prefer self-directed training. Around two-thirds would like to get more training on technical skills and managerial/leadership skills. The skills Specialists think they need to improve the least are computer skills, presentation skills, and interpersonal skills. It is important to note that external training options are relatively expensive, and KNL

-19and Unit managers must consider tradeoffs and cost-effectiveness in weighing these options.

IV. SUMMARY AND RECOMMENDATIONS 4.1

Since 2008, KNL has made significant progress in developing and consolidating a system for organizing, coordinating and managing the Bank’s training program.

The new system improves the efficiency of the Bank’s training program by addressing the fragmentation of training initiatives and funding sources previously prevailing in the Bank. It seeks to provide transparent and verifiable mechanisms for training budget allocations across and within organizational Units.

4.2

Among specific steps undertaken, KNL has developed a system to disseminate and store information on training activities, which is a key ingredient for a successful and systematized training program. To date the system is underused, however, as staff use it mainly for registration purposes. Quality control mechanisms for training events have been put in place, which are useful to assess the immediate reaction and perceptions of training participants. But there is limited evidence of them being used to improve the quality of training and learning or to measure gains on staff competencies.

4.3

KNL has also made significant progress in defining a formal process for Units to prepare plans for annual training activities. KNL assists the Units in this process to ensure that a robust analysis of the training needs is done and effective learning solutions are proposed. Remaining challenges perceived by staff include the lack of staff consultation about training needs and the weak link between career development and training. The amount of resources available determines the kind and quality of training provided. Limited funds reduce the opportunities for staff to receive individual training within and outside the Bank.

4.4

While staff are satisfied with KNL’s logistical and facilitation support for events, improvements in this area are possible. In particular, staff would like KNL to provide more help in characterizing the audience and defining the learning methodology to better adapt the presentation techniques and learning materials to specific audiences. With respect to the content of events, most of them agree that Banks’ Units should keep defining it as they have done until now. More than half of interviewed staff are satisfied with the quality of the content of technical courses, although they would like them to be more specialized.

4.5

On the utility of training courses, staff think that the New Employee Orientation course is not useful, and those interviewees transferred to a country office had not received an induction course on local systems. KNL has financed Retreats, though not considered as training, and some Units use most training funds for these events. In general, staff have a strong positive opinion of the value of Retreats, which are considered to be useful for strengthening team work abilities and

-20opening business opportunities, However, they are not necessarily geared toward improving staff competencies.

4.6

To improve the effectiveness of training for technical operational staff, this evaluation recommends that IDB Management (including KNL in coordination and collaboration with other Bank Units), should: a.

Continue to upgrade KNL’s online system to ensure that course descriptions and course feedback are complete and easily accessible (integrating eventually with Optima as appropriate);

b.

Reinforce the requirement that managers and staff in both HQ and field offices discuss and agree on individual training needs and plans annually;

c.

Establish a clear distinction in terms of objectives and sources of funding between technical training activities and activities (such as Retreats) focused primarily on team-building, to avoid crowding out the former by the latter;

d.

Explore ways to expand training options to meet specialized professional needs, including focused technical courses and costeffective external training;

e.

Strengthen the content of new employee orientation (perhaps through shorter and more frequent offerings) and induction for staff transferred to field offices;

f.

Continue to explore new ways to assess and ensure the quality of training events, including advising trainer on effective communication techniques.

BIBLIOGRAHPHY IDB (2005). “New lending framework. Evaluation report and recommendations. Final version,” Document No. GN-2200-13.

IDB (2006a). “Evaluation of the IDB’s Studies,” Document No. RE-323.

IDB (2006b). “Realignment of the Bank to take on its strategic challenges,” Document No. GA-232, Confidential.

IDB (2008). “Knowledge and Learning Strategy proposal (2008-2010),” Document No. GN-2479.

IDB (2010a). “The Country Studies Initiative and its effect on the Bank’s Knowledge Strategy,” Document No. RE-372.

MANAGEMENT’S RESPONSE TO THE REPORT, “AN EVALUATION OF ONE PILLAR OF THE IDB’S KNOWLEDGE AND LEARNING STRATEGY: TRAINING ACTIVITIES FOR IDB OPERATIONAL STAFF” (RE-401)

FEBRUARY 2012

Management’s Response to the Report, “An Evaluation of One Pillar of the IDB’s Knowledge and Learning Strategy: Training Activities for IDB Operational Staff” (RE-401) Management welcomes the report prepared by the Office of Evaluation and Oversight (OVE) and will used it as an input in developing the new phase of the Bank’s Knowledge and Learning (K&L) Strategy.

OVE’s report acknowledges the “significant pr ogress … in developing and consolidating a system for organizing, coordinating, and managing th e Bank’s training program ,” making the program more efficient by “add ressing the fragmentation of training initiatives and funding sources previously prevailing in the Bank.” Indeed, under the previous system, each departm ent budget was allocated without any institu tional guidance or clear rules regarding resource use or accountability for results. Consequently, supervisors were able to use the funds at their discretion for a variety of activities, including individual training a nd career development activities such as graduate or postgraduate degree cou rses. Following the ap proach taken with th e realignment and reaffirmed by the K&L strategy, the scarce re sources allocated to these activities were targeted to those activities essential to the Bank’s efforts to serve the countries, according to decisions made by supervisors based on inst itutional work plans. In other words, the Bank has moved from a system wherein fungible learning resources were managed on an ad hoc basis as an additional economic benefit for individuals to a strategic system that invests scarce training resources to address institutional priorities based on its work with the countries. This is the b ackdrop against which several of the staff responses to the OVE survey should be understood (more on this later).

In addition to the above, further context is needed to pro vide a frame of reference for OVE’s evaluation and Management’s comments: 1. According to OVE’s report, the evalua tion’s main objective is to “identify strengths and weaknesses in the im plementation of the Bank’s training program for technical staff engaged in providing opera tional services to clients.” Hence, as acknowledged in the intr oduction, the evaluation does not address other dimensions of the K&L strategy. The ev aluation does not cover (a) training for staff not working directly with clients; (b) the knowledge management activities, products, and tools other than tr aining in the form of courses; 1 and (c) external training and knowledge management for participants from the borrowing member countries. Furthermore, the content and focus of the information services provided by the Library are touched on only very briefly.

2. With this narrow focus on staff in the Bank’s operational departm ents, OVE’s survey limits the sample even further to just one segment by covering only grade 4 and 5 pr ofessional staff in the Vi ce Presidency for Countries, the Vice Presidency for Sectors and Knowledge, and the Vice Presidency for Priv ate

1

The best pract ices for adult professional learning recommend integrating these two lines of action (knowledge management and training), making it possible to effectively impact both mutually reinforcing dimensions through knowledge and training programs, products, and tools (see the section entitled “Knowledge and learning strategy—conceptualization and scope” in document GN-2479).

-2-

Sector and Non-sovereign Guaranteed Op erations. The results therefore do not reflect the opinions of staff at other gr ades or of supervisors, who, as was mentioned, formulate the departm ental K&L plans that align the investm ent of resources and staff time in K&L activities based on the organization’s need for the acquisition, sharing, dissem ination, and use of knowledge. The surveys often reflect the individual perspectives and priorities of the respondents based on their own career development, which, according to the strategy and current policies, is not included in K&L activities. W hile training activities did in the past allow f or the possibility of supporting career de velopment and the earning of university degrees and certificates, as p reviously mentioned, the current K&L policy provides for learning activ ities that lead to individual intellectual capital formation only if they contribu te to improving the organization’s performance in serving the countries.

3. Lastly, it is important to distinguish between K&L activities, which generally are a Bank-wide responsibility, and th e specific functions of the Knowledge and Learning Sector (KNL). For ins tance, the induction of new em ployees falls primarily to the hiring units, the Human Resources Department (HRD), KNL, and the country departm ent in the case of staff assigned to the Country Offices.

Another area in which KNL’s activitie s dovetail with the institution’s much broader scope of functions is information technology, a keystone of K&L. With these clarif ications, we view the re port as po sitive, and will take its recommendations into account in preparing the new strategy. In this connection, we offer the following comments: Recommendation (a): Management concurs with th e recommendation regarding the need to continue to upgr ade the online m anagement and infor mation system (KNLSystem). It is worth noting that this system was initially developed by KNL as a temporary solution, to provide the Bank w ith a platform for planning and costing activities, products, and events and to transparen tly allocate the bu dgetary resources centralized by the realignm ent and m anaged via the K&L Fund, while com prehensive solutions were developed for the institution as a whole (Optima). Given the need to account for the use of resources, however, KNLSystem turned into a m anagement information system that serves not o nly to plan but also to a llocate resources by activity and by product, linking them to each unit’s learning products and programs. It also serves as a participant registration and inform ation platform and as a database containing the training history for all staff, along with a de tailed description of knowledge and learning activities and the knowledge produc ts associated with course s and other Bank activities, while providing for effective, visible evaluations of all courses of four hours or more.

In fact, KN L is redes igning the In tranet page to m ake access to information in u serfriendly formats even easier. KNL invests s ignificant resources to pro vide this access, since the KNL database and inform ation system were developed and are m aintained by the department’s team. This is why KNL is on the lookout for possi ble integration with Optima solutions or other Bank investments that might lower the costs otherwise incurred in developing and maintaining its own systems.

-3-

Recommendation (b): According to the report, res pondents expressed concern over the weak link between the s taff’s career and pr ofessional advancement and needs identified as part of the unit’s training plan. It ther efore recommends reinforcing the requirem ent that managers and staff in both Headquarter s and Country Offices discuss and agree on individual training needs and plans annually.

The staff’s aspiration s are certain ly valid, but as indicate d earlier, it is im portant to separate the staff’s ind ividual professional and career ad vancement needs from the individual intellectual capital formation that is needed to enable the organization to better serve the countries. The fact that these two objectives occasionally coincide does not alter the substance of the institutional K &L agenda and stra tegy objectives, which cente r on promoting knowledge activities relevant to the organization, not the individual.

The K&L strategy m ade clear that to firm ly establish a knowledge and learning agenda for the Bank and allocate lim ited resources efficiently an d transparently, an an nual planning process based on the ultim ate goal of serving the development needs of the countries would need to be introd uced. With the aim of focusing on the ins titution’s thematic priorities and m aking good use of sc arce resources, the st rategy set f orth that K&L plans would include only: (1) knowledge and learning activities that m et demands directly associated with the Bank’s activitie s, and (2) topics arising from the for wardlooking identification of knowle dge areas in which the inst itution needed to create comparative advantages. Depending on the case, those demands could be based on: 9 The priority policies and in itiatives adopted by the Boar d of Executive Directors as well as the decisions and prioritie s of senior m anagement (“top-down” demands), and 9 The Bank’s priority topics identified by both country and sector operational units and by nonoperational functional units, a nd, when appropriate, by the Bank’s counterparts and partners in the bo rrowing member countries, which are aggregated by the operational depart ments and prioritized (“bottom -up” demands).

The knowledge and learning plan is the key to strategy im plementation in each unit, and all these plans taken together make it possible to identify a learning agenda based on the organization’s priority dem ands. Since 2008, KNL has prom oted this approach to identification of priorities and strategic planning of activities, working with th e units on priority needs and gap analyses, which are the foundation of the K& L plans and need to be addressed in order to adhere to the annual (and sometimes multiyear) work programs.

This alignment has led to the iden tification and prioritization of collective and individual needs, both unit-specific and cross cutting, resulting in a broad array of cost-effectiv e programs and products that m eet quality standards and are organized and funded for the purpose of updating knowledge and building capacities.

Individuals learn and receive training through various m echanisms that enable them to develop basic and technical competencies, and regardless of the m echanism, when its inclusion as a priority within the K&L plan calls for K&L Fund resources to be allocated, the impact of that investm ent on the indivi dual’s and the organization’s perform ance in contributing to the Ban k’s development agenda m ust be estim ated. Viewed f rom this

-4-

angle, individual career advancem ent will be the result of outstanding perform ance (enhanced by K&L actions, am ong other thin gs) but not the reason for selecting an individual to take part in training activities. A plan is not the sum of individual needs but the result o f management’s prioritization of the departm ent’s or unit’s collective and individual needs. Hence, the approved plan s are an expression of the dem ands of the units, not of the managers, as indicated in the evaluation. That said, Management agrees with OVE’s recommendation to ensure not only the broadest participation by staff in the process of identifying and prio ritizing knowledge and training n eeds, but also the tim ely communication of management decisions reflected in the plan with a view to carrying out the unit’s strategic program.

Individual professional advancement and career development is a di stinct and separate objective that has been included under the IDB’s Hum an Capital Strategy and is a responsibility shared by HRD and the superv isors; its im plementation requires specific guidelines and its m anagement an a dditional appropriation of budgetary resources. The K&L Fund lacks the resources to fund individual career development activities and at the same time address the updating and capacity-building needs deriving from the challenges of the organization’s work plans.

Recommendation (c): Management supports, and has applied, OVE’s suggestion to draw a clear distinction betw een technical training activ ities and a ctivities focused primarily on team-building in the abstract. On this poin t, the term “retreat” can refer to various types of activities that should not be confused. KNL does not support retreats for team-building in the ab stract—only when they serve an institu tional objective. F or example, the tran sition from the previous or ganizational structure to a m atrix structure and a results-based managem ent system has been one of the realignm ent’s most significant challenges. Working in a m atrix organization necessitates greater cooperation among people and between units, since no part of the system is self -sufficient. The understanding of how t o work in a m atrix framework took up a m ajor part of the new management and technical team s’ discussions from 2008 to 2010. KNL supported strengthening of the country team s, through workshops (misnamed “retreats”) held at the Country Offices to that end, based on the premise that learning to perform as a member of a matrix team was a key, basic competency recognized by comparator institutions in their learning strategies and funded in their budgets.

The “sector weeks” are another example of collective activities. With the creation of the sectors, sector technical capabilities we re merged, bringing the form er knowledge communities under single leadership. This boosted the organization’s productivity, given the improved coordination and the increased s ynergies resulting from the clustering of sector specialties, but it also called for new knowledge management approaches to reduce fragmentation and stren gthen the specialized knowledge resources of Bank staff at Headquarters and the Country Offices alike.

The sector weeks enab led all t echnical staff in an infrastructure sector to exchange information and knowledge, provided learning opportunities for addressing new technical and operational challenges, and allowed knowledge to be shared with cou ntry counterparts, experts, and o fficials, all of which has a ppreciably enhanced the Bank’s value added as a development partner.

-5-

If the recommendation is to not support “retreats” in general, then it is consistent with the KNL practice announced in December 2011 for the funding of 2012 plans. If, however, the recommendation is to be interpreted so as to exclud e the types of collective events mentioned above with a view to freeing up scarce resources to fund individual training activities, then this would elim inate an important and successful K&L com ponent, and one in which employees from the member countries participate.

Recommendation (d): Management supports OVE’s sugge stion to explore w ays to expand training options to meet specialized professional needs, based on the requirements of the organization and the countries. External training is a dominant feature of the plans of several Bank departm ents and units. For the Information Technology Department, the Legal Department, the Office of the Executiv e Auditor, the Finance Departm ent, and other service departm ents, external courses offered by specialized institutions to bring individuals’ knowledge up to date in terms of IT systems and applications, regulations, or financial models account for a significan t percentage of the time and reso urce investments in their respective plans. In 2011 the num ber of participants trained in programs sponsored b y outside institu tions grew to 811 (20% hig her than in 2010).

Courses and workshops were also organized at the Bank with specialized experts. T he average cost of enrollment in indiv idual external training program s was approxim ately US$1,500, without counting travel or per diem . The average cost of enrollm ent in highly specialized executive courses on risk management and economic development was three or four times the average, which explains why it is difficult to extend this training option for the sector and country teams. KNL will encourage the adoption of more cost-effective mechanisms, such as online train ing. Internal conditions need to be created wherein the time required for self-learning and participatio n in virtual courses is recognized. As evidence of this challenge, in 2011 only 48 Bank employees took pa rt in the virtual courses offered by the Inter-American Institute for Economic and Social Development in the region (3.5% of total par ticipants). Encouragingly, 85% of them were from Country Offices located in Group C and D countries.

Recommendation (e). KNL and HRD believe that the ne ed to improve the process and content of the new e mployee orientation progra m is urgent, particularly at the Country Offices and for those who rota te between Headquarters and the Bank’s offices in its member countries. Management therefore fully agrees with priority highlighted in OVE’s report to strengthen that program , but disagrees with the re commendation to shorten the offerings. The nature and requirem ents of working at the Bank necessitate an orientation program that includes all the pro cesses necessary to integrate new employees fully into Bank life. Orientation has two phases: initial orientation and ins titutional assimilation/acquisition of specific ins titutional skills. Thro ugh 2010, KNL’s program focused on the first ob jective: orientation aim ed at sharin g the institution’s m ission, objectives, strategies, and values, and setting expectations regarding codes of conduct and ethics and performance standards. Beginning in 2011, the orientation program has been strengthened by the addition of an operati onal training program , including operational work processes and knowledge of the policies, systems, procedures, and tools for project management, to shorten the learning curve a nd attain maximum productivity in less time.

To do this, the program ’s duration was exte nded from two to five days. Particular emphasis was placed on the participation of Country Offic e staff, which also tran slated

-6-

into higher travel and per diem costs. Nonetheless, we believe that two additional changes are needed in order to achi eve a truly effective onboarding program: an action plan that establishes effective coordination between HRD, KNL, the contracting departm ent, and the country department in the case of staff assigned to a Country Office, based on clear, specific guidelines and responsibilities for each unit, and the developm ent of better tools for supporting the orientation program for Country Offi ce employees. The Hum an Capital Strategy’s action plan provides fo r the review and im plementation of a new employee orientation program that incorporates the above changes. Once the program has been reviewed and redesigned, it w ill be poss ible to estim ate and weig h the budgetary impact of its implementation.

Recommendation (f). Management agrees with OVE’s recommendation . KNL recently finished designing a user-friendly report that can be sent to everyone involved in a training activity, to diss eminate the results of the “Level 1” evaluation s (i.e., feedback provided by participants at th e end of the ev ent). We trust th at this repo rt will be conducive to proactive, joint quality management of training activities.

At the same time, KNL will seek ways to ensure that all its liaisons play a role in quality management, including in thei r activities a rigorous analys is of the learning objectives and the alignm ent of those obj ectives with the event’s age nda and, where appropriate, providing advice on effective training and communication techniques.