EMT Guide by EMT Guide

Electronic Monitoring Tool (EMT) Guide Conditions for a successful implementation of an EMT in your organisation.

Thank you! First of all, we are deeply grateful to all the NGOs who have contributed in the development of the guide, which was a very bottom up process. The content in this guide is based on a trajectory we went through with a dozens of NGOs. We would like to thank all the NGOs who participated in the process and gave us great insight and valuable input. Thank you!

ÂŠ 2015 ngo-federatie & ACODEV February 2015 Brussels Project conducted for the ngo-federatie & ACODEV by i-propeller

Contents Introduction 4 1. How to use the guide 5 2. Monitoring data: EMT in a broader context (Phase 0) 6 Monitoring data 6 Monitoring Software 7 Typology of ambitions 7 A strategic choice 8 Insights from the field 9 3. A phased process 10 Three phases 10 Implementing change 10 Structure of the guide 11 4. Preparation Phase (Phase 1) 12 Vision 12 Importance 13 Plan 17 Resources 18 Competences 22 5. Configuration Phase (Phase 2) 23 Vision 23 Importance 23 Plan 25 Resources 26 Competences 27 6. Use Phase (Phase 3) 29 Vision 29 Importance 29 Plan 29 Resources 31 Competences 31

Introduction The ambition of this guide is to help and support organizations that have not yet implemented an Electronic Monitoring Tool, by providing an overview of the thinking and doing processes that you go through to arrive at clear and actionable insights into the preparation, implementation and use of an EMT tool in your organization. Consider the guide as a roadmap that can help you in effective EMT planning. Before you start EMT, you should have thoroughly developed the PME system in your organization. Our EMT project is not aiming for a perfect total solution, as you would expect from systems such as ERP, SAP or other integrated systems. EMT here is focused on Monitoring. This guide is the result of a two year term project: after we mapped in 2013 the electronic monitoring needs of different NGOs and searched the market for appropriate tools that correspond to these needs, we held in 2014 a practice lab where identical cases were presented in the 3 selected software tools. End of 2014, 3 NGOs decided to implement software tools in their organization. All of these lessons and experiences are integrated in this guide. Our first EMT connection is primarily related to quality. The federations continuously support and enhance the quality of work in development aid. Our second EMT connection is transparency. If we monitor the progress we make in the realization of our aims, then we can collect data and produce narratives. We can export (a selection of) these data and narratives, and publish it to targeted audiences (e.g. in the common IATI language). We hope you find this guide useful. On the websites of the federations*, you‘ll find more information about the exchange platforms we have organized specifically on Electronic Monitoring Tools. Enjoy reading! ngo-federatie & ACODEV

* Please note that you need a username and password to enter the website of ACODEV. “DGD financed the EMT study because we are convinced that such EMTs may contribute to the further professionalization of the NGO sector. An EMT may indeed help organizations to better monitor their programs and projects, and offers paths to also simplify the reporting to DGD and other donors. EMT can be a step towards more transparency on the results of development cooperation, a challenge we collaborate on. We wish you the best of luck.” Katrien Degrauwe Head of department Civil Society Asia and Latin America Directorate General for Development Cooperation

1. How to use the guide Objective This guide is a practical guide that helps you to guarantee conditions for a successful implementation of an EMT. Reading this guide will give you knowledge and insight in the preparation, configuration and use of an EMT, combined with practical handles which will help you with the implementation of an EMT of your choice. The EMT guide is an instrument designed for NGOs. Often, the implementation of an EMT is led by one person or a working group within the organization. This tool should serve as a guidance to help this person or group in taking the lead. The person(s) in lead are responsible for bringing the right people together for specific exercises in the guide. The focus of this guide is not on organizations who do not know yet what to monitor, however, going through this guide might be a good exercise to help your organization gain insight into the EMT software and to encourage your organization to reflect and define parameters. E-book This guide is an e-book, a visual and interactive pdf file with lots of links to external media. Through the various chapters of the guide, you will see some guiding icons1. These icons inform you about the type of content so you can easily go through the pages. Some of the icons include a link to external media. Legend Exercise Questions Tips and tricks Food for thought NGOs speaking Watch a movie

includes link to external media

Good reading includes link to external media Link to website

includes link to external media

The icons are based on creations form Martin Delin, Joe Harrison, Arielle assoulline-lichten, David Vickhoff, SuperAtic LABS, Roy Verhaag and Brandosour.us from the Noun Project. 1

2. Monitoring data: EMT in a broader context (Phase 0) Before we go through the different phases to implement an Electronic Monitoring Tool (EMT), let’s take a step back. In this chapter we take the time to look at EMT in a broader context. Questions • What do we understand by ‘monitoring’? • And why should an organization implement ‘monitoring software’?

Monitoring data “Monitoring is a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.” 2 The figure below gives a clear overview of how monitoring data is part of a bigger picture, and has a central position in a virtuous cycle of planning, monitoring and evaluation. In this guide we will focus on the monitoring part, using electronic monitoring software, but the three elements of the cycle are inextricably linked.

Planning

Monitoring

Evaluating

"Theory of change"

A continuous process of systematic data collection on specific sectors. Is the project doing the things 'right'?

Systematic and objective assessment of a (finished) project, program or policy; and its design, implementation and results.

progress according to implementation plan.

Is the project doing the 'right' things?

"We believe by doing X, we will reach result Y"

Intended goals (outcomes/impact)

Monitoring, part of a virtuous cycle

Adapted from Ten steps to a results- based monitoring and evaluation system, Kusek, J.Z. and Rist, R. C. The International Bank for Reconstruction and Development / The World Bank, Washington, 2004. 2

Closely related to the cycle is the overview of guidelines for the description of an EMT system that the ngo-federatie wrote, based on a study from Nathalie Holvoet and Dirk Van Esbroeck.

Monitoring Software Organizations work with multiple donors, each with their own needs for specific information and formats. This can be a challenge because information is collected continuously, stored and manipulated by different people in different locations. The search for the most recent version of a particular spreadsheet or new template leads to unnecessary loss of time, inconsistency or simply the lack of crucial data. The need to spend less time on reporting arises, as stakeholders use slightly different indicators in different formats. The use of an EMT allows employees to enter results and indicators only once, at the time that activities are conducted, and stakeholders can access the data at the time they need it. With an EMT you can easily refer all applicants of (project) data to one central place. Creating and modifying tables and graphs is easier and more user friendly. Exporting to Word, PowerPoint or Excel remains possible. Gaining insight into the evolution of a (multi) donor program is simple. An Electronic Monitoring Tool provides several people simultaneously access in a unified way to different data sources in a centralized system. The organization itself makes decisions regarding access to information: from what role each user has to what access to data is given (e.g. limited to just read, add, or change). Partners can be given access to program results from other partners or just their own results. Interaction with the EMT ranges from simple input of data to complex reporting of results. On condition of use of a centralized EMT, patterns, trends and reports can be generated easily from these sources. The user interface of the EMT can be customized in a way that the user only sees and enters data that are relevant to his/her specific role in the process. The use of an EMT allows an organization to view the organizational structure and thus visualize information through projects in a more efficient way. The fact that EMT software is developed by people with a background in the development aid sector ensures that the people behind the software have insight in your organization and are well-placed to understand you. Moreover, the strength of an EMT lies in the fact that it pushes your organisation to think outside of its normal way of operating. The EMT software challenges you, obliges you to think about the vision and structure of your organization.

Typology of ambitions It is remarkable, and perfectly legitimate, that the goals that NGOs are aiming for with the possible introduction of an EMT vary; not so much in terms of principles, but rather in terms of accents or priorities. This implies differences in the way optionally testing of the EMT should be best accounted internally. Each software has its own focus: accountability, knowledge acquisition or a combination of the two. The exercise below is a first exercise to gain high level insight in the requirements your origanization has. 7

Depending on your answer, other software may be interesting for you. Please note that your level of ambition may evolve over time. If you know your organization is going through an evolution, then go for a program that can handle a lot. Exercise 1: Take a look at the table below and position your organization in terms of ambition. Which typology fits your organization the best? What is the consequence for the EMT?

Time Org. culture Pragmatism

TYPOLOGY AMBITION

CONSEQUENCE FOR EMT

EXAMPLES

1. External accountability: Most important factor is external accountability to e.g. donors, partners, media...

Thanks to the EMT, the organisation can better fulfill the (minimum) monitoring requirements of donors

Access, Excel, Open ERP, Smartsheet, Word

2. Knowledge acquisition: Most important factor is knowledge acquisition, gaining better insight into its own processes, efficiency and effectiveness of the operation

Thanks to the EMT, the organisation gets better insight into its own processes, what it's doing, good/bad, where there is a need for intervention

Akvo, Outlook, Open Project, SharePoint, Wiki

3. Multi objective: No priorities, everything is equal + learning network, positioning

Flexibility. Minimum equipment must excel in at least the two criteria above.

Actfact, Activityinfo, Aidimpact, Newdea, Sigmah

Figure: Typology ambition and consequences for EMT

A strategic choice Three convergences (Triple C ) explain the strategic choice to invest more in smart monitoring systems functional, horizontal and geographical convergences3. Functional Convergence Functional convergence implies standardization and more players on the market who take up multiple and complementary positions. This results in a more competitive market. 3

Adapted from â&#x20AC;?The World is Flatâ&#x20AC;? written by Thomas L. Friedman. 8

Example: The Paris Declaration The Paris Declaration (2005) is a practical, action-oriented roadmap to improve the quality of aid and its impact on development. It gives a series of specific implementation measures and establishes a monitoring system to assess progress and ensure that donors and recipients hold each other accountable for their commitments. Horizontal Convergence People and organizations work together across borders. Example: Be-cause Health Be-cause Health is an informal and pluralistic platform, which is open to institutional and individual members. The platform connects members from different categories: (i) Belgian academic institutions, services, schools for public health, (ii) Belgian medical (and related) development organizations, NGOs and study bureaus, (iii) government services involved in medical development cooperation and international health care and (iv) organizations and associations who do not necessarily have development cooperation as a priority, but whose mandates overlap with international health care. Geographical Convergence Geographical convergence including following trends: â&#x20AC;˘ Physical distance is losing importance. â&#x20AC;˘ Globalizing market. â&#x20AC;˘ Information is everywhere.

Insights from the field Curious why other NGOs want to implement an EMT? Read the motivations of three organizations below. Organization 1 promotes medical care for the most disadvantaged people in Africa, Asia and Latin-America. The organization manages different projects which are financed by different donors, a complex situation that asks for professional follow-up. With the implementation of an EMT, the NGO aims for a more efficient follow-up of the different ongoing projects and programs in their organization. In addition, the implementation of an EMT is also part of the quality improvement of monitoring and evaluation within the organization. Organization 2 aims to improve the quality of life of children in Africa, Asia and Latin-America. The organization currently manages a portfolio of roughly 25 projects funded by different sources. In order to manage its operations, the NGO has completely revised its planning, monitoring and evaluation (PME) system. A detailed project tracking path has been developed with related tools. The new PME system also includes the need to document all steps in an electronic management system to provide a database for each project in order to analyze and process data, to have an alert process, an indicator management system... Given the increasing number of projects and the revised PME system, the organization decided to acquire an EMT. Organization 3 is active in approximately sixty countries to organize development projects for people with disabilities. A division of the organization abroad has been using an EMT for already two years now. The NGO wants to implement the same tool and hopes to improve the monitoring of all the projects they have set up.

3. A phased process Three phases The implementation of an EMT requires a lot of thinking and preparation. An organization, who wants to implement EMT software, typically goes through three phases: the preparation phase, the configuration phase and the use phase. The time you need to go through these phases varies for each organization.

Phase 1 Preparation

Phase 2 Configuration

Phase 3 Use

A phased process

Implementing change Introducing new monitoring software in your organization implies change, and implementing change is a matter of different elements. Next figure gives insight in what is needed to make change successful, and what happens if vital elements are missing.

Vision

Importance

Plan

Resources

Competences

Change

Importance

Plan

Resources

Competences

Confusion

Plan

Resources

Competences

Resistance

Resources

Competences

Chaos

Competences

Frustration

Vision Vision

Importance

Vision

Importance

Plan

Vision

Importance

Plan

Resources

Figure: What makes change successful?4

Adapted from â&#x20AC;?Verandering. Wat maakt succes?â&#x20AC;? by Guido Vermeeren. 10

Anxiety

Structure of the guide This guide will lead your organization respectively through the preparation, configuration and use phase. In each phase we will discuss the 5 elements needed to make change happen: vision, importance, plan, resources and competences.

Phase 1: Preparation

Phase 2: Configuration

Phase 3: Use

vision

importance

plan

resources

competences

Figure: structure of the guide Going through the 3 phases, we want your organization to be able to answer the following questions: Questions What are the needs of your organization? (Phase 1) How can your organization prepare optimally for the implementation of an EMT? (Phase 1) How to configure the software to match your needs? (Phase 2) How to ensure the software is integrated in your daily operation? (Phase 3)

4. Preparation Phase (Phase 1) The first phase is the most important one and requires a lot of thinking. Once you have went through all the steps of the preparation, your organization is ready for concrete action, i.e. configuration and use of the EMT. Questions What are the needs of your organization? How can your organization prepare optimally for the implementation of an EMT?

Vision Long term vision What does your organization want to achieve in the long term with the implementation of monitoring software? In the preparation phase, communicating the vision of your organization is a way to get different stakeholders on board. The buy-in of different stakeholders requires an investment. What is the added value you are aiming for, not only in the first year, but also the period thereafter? These questions are situated in the same tendencies we notice in the development world. Demonstrating impact and progress are increasingly important; transparency is key. Example: IATI The International Aid Transparency Initiative (IATI) is a way to make information about aid spending easier to find, use and compare, by using a common standard for publishing information. Exercise 2: What is your ambition? Who are the stakeholders you want the ‘buy-in’ from? This exercise will give your organization insight in the level of ambition. Please follow the next steps in order to fill in the table. 1. What is the agenda of your organization? Which objectives are you aiming for with monitoring? E.g. accountability, decision support, knowledge acquisition and dissemination, positioning… 2. Who is/are the target group(s) for each objective? E.g. internal (own staff + volunteers), Board of Directors, beneficiaries, partner organizations, policymakers, media, experts, financiers… 3. Divide 100 points on the objectives, indicating the relative importance. What are the priorities for your organization? 4. Take a look at the target groups and describe in the fourth column what the information is they would like to know. What is important for them? 5. Double check: is everything in line with the values of your organization? E.g. participatory approach, capacity building…

Your perspective, your focus

Other perspectives

Objective

Target group

Weight

What does your target group want to know?

100

Tip! Everyone should see and understand the value and purpose of the tool. ‘Goodwill’ is crucial. According to some NGOs this is a challenge for partner organizations and development workers in the south. “Make sure that the top management in your organization, head office and field level, supports the project (also with appropriate resources, including IT) and that it fits into the organization’s strategy. This is the first precondition for success.”

Importance Level of monitoring Let’s have a look at today’s operations. With the exercise below we want to take a look at the current monitoring efforts. Exercise 3: How is today’s monitoring organized at the different levels? Use the tables below to map the current monitoring situation. For each level, answer the following questions: 1. What is monitored? 2. Who is responsible? 3. Which tools are used? 4. How frequent?

What is monitored?

At the NGO level in the North Who is responsible? Which tools are used?

How frequent?

At the NGO level in the South Who is responsible? Which tools are used?

How frequent?

At the level of partner organizations Who is responsible? Which tools are used?

How frequent?

At the level of beneficiaries Who is responsible? Which tools are used?

How frequent?

What is monitored?

â&#x20AC;&#x153;Preparing for an EMT makes you think about your logical framework. The tool provides insight into your organization with technology as an enabler.â&#x20AC;?

System requirements EMTs have a lot of functionalities. In this part we want to help you to stipulate the desired functionalities for any EMT software to meet the needs of your organization. The table below gives on overview of different system requirements in the following domains: functionalities, interface, monitoring, reports, software, training, helpdesk, and history. We identified these requirements based on needs analysis amongst a bunch of NGOs, starting from an existing list. Depending on the needs of your organization, you can divide these requirements in ‘must haves’ and ‘nice to haves’. ‘Must haves’ are requirements that your EMT must have in order to achieve the monitoring goals your organization has set out. ‘Nice to haves’ are system requirements that are not crucial to achieve your monitoring goals. Exercise 4: Which system requirements are ‘must haves’ for your organization? Which requirements are ‘nice to haves’ and which requirements are not needed? Go through the list and check the appropriate boxes. System Requirements Functionalities Operational monitoring Project management Managing Funding Accounting Interface Multilingualism Customizable dashboard Publication interface Consultation interface Multiple users Multi-level (partners, local offices, headquarter) Multi-projects Offline use of the software and synchronization when online again Monitoring Quantitative indicators Qualitative indicators Geo-local indicators Import of data (from e.g. bookkeeping systems) Export of data Integration with other programs Reports Customized reports Dynamic cross tables Export word / excel Software Open-source license Training e-learning 15

Must haves

Nice to haves Not needed

System Requirements Must haves train-the-trainer End user training Help desk Service level agreement Response time History of supplier Same customers and users, assessment of the continuity of service Other?

Nice to haves Not needed

For those who are interested in a more technical list of the system requirements, another list is available. During the trajectory, we organized a workshop to gain insight into the ‘must have’ requirements for NGOs. The list below is a list of 15 ‘must haves’ that were selected by more than 50% of the 10 participating NGOs.

Top 15 must have requirements 1. Create or change the activities and their indicators which are part of this database 2. Lock databases, activities or projects to prevent users to add or change data 3. History of data input 4. Export reports to Excel 5. Save reports 6. Define targets to be reached by your project 7. Offline use of the software and synchronization when online use is possible again 8. Pivot Tables cross all of the dimensions of results, including by activity, time period, partner or geography 9. Define the partner organizations who participate in this database/program 10. Add users or control their access level 11. Export data 12. Charts and graphs to summarize data for reporting 13. Custom report combination of tables, charts and maps 14. Share with other users 15. Language option in French

Tip! Monitoring and capturing activities is inherent in efficient EMT, but for the interpretation, an efficient representation of data in various graphical ways is necessary. Tip! Compatibility between different languages is desirable.

Plan Use the following checklist as a guidance to plan the preparation phase.

1. Determining leadership

2. Analysing needs

3. Analysing tools

4. Final choice

Figure: Roadmap for the preparation phase 1. Determining Leadership • Who can coordinate the implementation process? • Whose buy-in do you need? • Who are the important internal/external stakeholders? 2. Analyzing needs • What is your ambition? • What does your organization need? • Which system requirements do you need? 3. Analyzing tools • Which software is available? • What do they offer? • What is the cost? 4. Final choice • Which software do you purchase? • What is your timing? • What resources do you need? “Internal communication is highly important, even at this early stage and also for non-directly involved colleagues.” “Put expected advantages and improvements forward, but do not hide foreseen difficulties and build solutions together with the main actors, namely the project managers.” Tip! Can you organize a joint-purchase with other NGOs? Tip! Keep an eye on the Quality fund from ACODEV and the ngo-federatie for new calls.

Resources In this part we will discuss three important kinds of resources: software, time and financial resources. Software Comparing system requirements A very important question is which EMT software is the most relevant one for your organization? The market of monitoring software is a fast growing market; and the products vendors are offering are quickly updated and expanded. In the trajectory we did, we started with a longlist of 15 vendors. After an intensive selection period with self-scoring, questionnaires, validation in demo environments and testing with NGOs, we ended up with three software solutions: ActivityInfo, Newdea and Sigmah. Different dimensions were included in the trade off: functionalities, price, usability, innovative nature and capacity of the vendor (continuity of quality). The table below shows an evaluation of the three chosen monitoring tools we made in December 2013. Please not that this table is a snapshot, made in December 2013. Updates that have occurred since then, are not included in our analysis. Even if you donâ&#x20AC;&#x2122;t want to work with these three tools, the comparison gives you a good idea of the elements you need to consider in the decision process.

Functionalities Operational monitoring Project management Managing Funding Accounting Interface Multilingualism Customizable dashboard Publication interface Consultation interface Multiple users Multi-level (partners, local offices, head quarter) Multi-projects Offline use, synchronization Monitoring Quantitative indicators Qualitative indicators Geo-local indicators Import of data (compa, â&#x20AC;Ś) Export of data

ActivityInfo

Newdea

Sigmah

v x x x

v v v x

v v x x

FR-EN

FR-EN-ES (+ other )

v web web v v

(FR and PRT coming soon) v web web v v

v v

v v v v v

x web web v v

Integration with other programs Reports Customized reports Dynamic cross tables Export to Word / Excel Cost License Financial model

Cost of use (hosting, technical supportâ&#x20AC;Ś)

Specific developments (integration of other tools, specific functionalitiesâ&#x20AC;Ś)

ActivityInfo v custom development

Newdea v

Sigmah ?

v v

v in development (2015) v

open-source software as service

closed source software as service

Free: up to 10p; Level 1: up to 75 users, 4950 USD/ year; Level 2: up to 150 users, 9950 USD/ year; Level 3: up to 1000 users, 14500 USD/year

Up to 6 users: 260 Euro/year; 7 to 26 users: 25 Euro/month; 27 to 100 users: 20 Euro/month; > 20 users: 7320 Euro; 100 users: 26880 Euro/year On demand: 1100Euro/day

On demand: 95 Euro/hour

open-source community management with funding for joint developments related to public funding Annual membership: (0.5*revenue^0.5), For an organisation with 10.000.000eur, this will be 1600eur

Logic of opensource community development, with collective governance

Training Estimated one day training one trainer at 650 Euro day, eLearning platform available

Estimated one day training at 1100 Euro + travel expenses

Estimated 1,5 day training at 1500 Euro one trainer + travel expenses

Estimated one week at 650 Euro day + travel costs. Customization of Activityinfo to the specific NGO is included in the training days

Estimated at 3 days implementation at 1100 Euro/day

Estimated at 2 days implementation at 500 Euro/day

Implementation

Help desk email, 1 business day

7/24, email, Skype, 1 forum, email, Skype, 1 business day business day

ActivityInfo Other History

Which users? Estimation of the continuity of the provided services?

Newdea

History Development Development supported by UNICEF initiated by World and UNHCR. Base of Vision limited users

UNICEF, UNHCR, MSF, EIP Foundation ++

especially American NGOs +++

Sigmah Development based on a preliminary version of ActivityInfo that integrates lessons from compass Qualité (Groupe URD). Development piloted by a user committee based on public funding (FR, EU). especially French NGOs +

Comparison of ActivityInfo, Newdea and Sigmah Organization 1 chose for ActivityInfo. The organization is mostly implementing health service delivery projects in rural areas, so they were looking for an easy to use interface for follow up of activities and indicators, corresponding to the logical framework. Also the possibility to work off-line was crucial as the organization works mostly in DRC with the health zones, which don’t have internet access on a regular basis. Organization 2 decided to choose for Newdea for the following reasons: • Project management according to the phases that correspond to their own trajectory • Complete follow-up of the project in terms of activities and results, and budget • Management of global indicators generated from specific indicators • Automatically updated project sheets for project presentations • Data transfer to / from the CRM system that the organisation has implemented Organization 3 decided to work with Sigmah, because the tool has already been used in other divisions of the organization abroad. Demo sessions In October 2014, ACODEV and the ngo-federatie joined forces to organize an EMT Practice Lab. The objective of this Lab was to facilitate the comparison between different tools, and to facilitate the decision making. ActivityInfo, Sigmah and Newdea were invited to present their software to all the members from ACODEV and the ngo-federatie. In order to adapt the demo sessions as much as possible to the needs of NGOs and to give them the possibility to compare, we described a fictitious North case and a South case, and asked them in advance to integrate the two cases in their demo sessions. To get an impression of the functionalities of ActivityInfo, Sigmah and Newdea, and to compare them with each other, you can review the demos of the Practice Lab. 20

Movie: ActivityInfo, North Case

Movie: ActivityInfo, South Case

Movie: Sigmah, North Case

Movie: Sigmah, South Case

Movie: Newdea, South Case5 Q&A Sessions To gather insight in the type of questions NGOs typically have regarding the implementation of an EMT, we would like to refer to the three Q&A documents that were made based on the Q&A session during the Practice Lab (October 9th, 2014) with ActivityInfo, Newdea and Sigmah: Read the Q&A session of Sigmah

Read the Q&A session of ActivityInfo

Due to technical issues, the North case from Newdea is not available. 21

Read the Q&A session of Newdea

Tip! Take time to properly assess the capacity of the Software Vendor (developer) including its reputation, its experience, size of its customer base. Why? There is always a risk that the developer goes bankrupt or will be dissolved, which may have potentially strong negative (harmful) consequences to further usability and extensibility of a customized tool. Time It was already mentioned before; the implementation of an EMT demands a lot of time and thinking. You need to invest enough time in the preparation phase, including the decision making of the selection of the software. Exercise 5: Go back to the ‘Checklist preparation phase’ and make for each activity an estimation of the time needed. Financial resources The financial efforts that an NGO is willing to and/or can provide regarding purchase, use and maintenance of an EMT differs greatly from NGO to NGO. The financial capability of an organization has an important influence in the decision making and therefore it is important to take into account all the different costs: • • • • •

Implementation cost Training costs Cost of use: hosting, technical support, fee for additional users… Specific developments: integration of other tools, specific functionalities… Potential costs for updates

Tip! The necessary initial costs to introduce an EMT can be reduced by for example purchasing the software with several NGOs, organizing joint training sessions, dividing the maintenance costs... Tip! Look for co-financing possibilities like the Quality fund from ACODEV and the ngo-federatie.

Competences Which competences are needed for the implementation of an EMT? Who can coordinate the process? The development has the highest success rate when it is followed up by someone with specific knowledge of the sector and knowledge of the internal processes. Often that person is someone internally. Keep in mind that he or she must be available during the configuration and testing period to answer questions and to do (corrective) actions where needed. Also think about how to involve the people in your organization who are responsible for the ICT. The sooner they are involved in the process, the better. “Involve future users, head office and field, in the initial needs assessment and the draft planning . Do not forget to involve the IT teams (if in-house) since the very beginning.”

5. Configuration Phase (Phase 2) Once your organization has gathered insight into what it wants and has chosen the right EMT software, the second phase can start: the configuration. Question How to configure the software to match your needs?

Vision Ensure the organization’s vision remains alive during the configuration phase. The vision will help you to shape the indicators. “Keep the participatory approach but stay within the framework of the Monitoring and Evaluation Policy used by your NGO, this will help channeling interventions and ‘framing’ the tool.”

Importance Prioritization An important part of the configuration phase, is designing the indicators you need and how to get results out of the data you collect. But before designing the indicators, we would like you to think about the most divergent projects that should be monitored in a consistent way. This exercise anticipates the choice of indicators regarding the type of projects. Please note that different projects do not always differ in terms of monitoring. This exercise is perfect for a group discussion. Exercise 6: Describe the two most divergent projects your organization would like to monitor. Is there a large diversity between the projects? Are they different in terms of monitoring? SPECTRUM Project A

Project B

“We signed a cooperation agreement for a research project of 3 years. In the framework of this specific cooperation, we have to look for the right indicators.” 23

Tip! Keep the configuration as simple as possible, i.e. no more compulsory steps or uploads than requested by your internal policies. You don’t need to monitor everything. Focus on projects that are important in size, volume and impact. Indicators Monitoring is an ongoing process of collecting data on specific indicators. The implementation of an EMT requires therefore a lot of thinking: thinking related to results (indicators) organizations are aiming for with their activities, and thinking related to the analysis the organization wants to perform based on the data collection. In a previous exercise in the preparation phase, we asked for a description of what your target groups want to know (see Exercise 2). The answers you gave to this question are a first step to specify indicators. Take a look at the table below. The table visualizes how to shift focus to results. The table is based on a well-known logical framework and goes a step beyond, allowing you to link input to results, giving insight into the cost-benefit efficiency of your work. Organization’s Planned Work Inputs

Organization’s Planned Results

Activities

Resources (capital, Concrete actions human) invested in of the organization the activity

Money, number of people

50k EUR invested, 5 people working on project

Development & implementation of programs, building new infrastructure… Land bought, school designed & built

Outputs

Outcomes

Impact

Tangible products from the activity

Changes, benefits, learnings, effects resulting from the activity

Attribution of organization’s activities to broader & longerterm outcomes Take account of actions of others, unintended consequences etc.

Number of people Effects on target reached, items sold, population e.g. etc. increased level of education New school build with 32 places

Places occupied by students: 8

New students with access to education: 2

Figure: Roadmap for impact If you base the configuration of your monitoring software on the different steps of the roadmap above (i.e. input, activities, output, outcome and impact), the EMT can make the link between work and results, which will allow you to gain insight in the cost-benefit efficiency and to better communicate about the effectiveness. Exercise 7: List in the table below the activities you would like to monitor. Describe for each activity the input needed and the planned result: what are the outputs, outcome and impact? Note: defining impact is a difficult exercise!

Inputs

Activites

Outputs

Outcomes

Impact

Look at the example below to see how you can link the logical framework of your organisation with EMT software. Logical Framework

ActivityInfo

Specific Objective (SD) Results (R) Indicator (Ind)

‘Catégorie’ ‘Formulaire’ ‘Indicateur’

Example logical framework and EMT software ActivityInfo Tip! Start with a manageable set of indicators to pilot test the software. Focus for example on a few top level indicators in all countries, and fine-tune afterwards with additional indicators for each country. Tip! Data security is a must for any reliable system. All three selected tools have robust user access level controls embedded. Consider which people need what access to which content (view, update, delete…).

Plan Use the following checklist as a guidance to plan the configuration phase.

1. Determining leadership

2. Analysing users

3. Validation indicators

Roadmap for the configuration phase 25

4. Train the trainer

1. Determining leadership • Who is responsible for the configuration? • How much configuration can be done by the organization itself? • Who is capable to do the configuration (link between needs and set up)? 2. Analyzing users • Who are future users? • Which profiles do they need? 3. Validation indicators • Which activities have priority? • Which indicators do you need? How to get them out of the software? • Who can validate the indicators internally? 4. Train the trainer • Who will train the future users? “Do not forget to keep the internal communication on a regular base.” Tip! Check with the vendor if there are any specific demands regarding preparation. Tip! Use the descriptions in the various fields to ensure what type of data is expected to be input. Use a number field if numbers are the expected input instead of text. To avoid confusion in a multi country environment ensure how thousand separators with comma or decimal points are to be handled. Tip! Ensure system settings warn users when data is imported with inconsistent values.

Resources Tool Vendors usually offer their product as a service. They can help you with the implementation and trainings. Take into account the cost that goes with it. Please remember that you are not the only NGO implementing monitoring software. Make use of the experiences and learnings from other NGOs who have already went through the configuration. Take a look at the online learning platform from the NGO Federation. Time Foresee enough time for the configuration phase. Close this phase with a train the trainer moment. Exercise 8: Go back to the ‘Checklist configuration phase’ and make for each activity an estimation of the time needed.

Financial resources Each monitoring software has its own financial model, whereby the cost of use varies. Costs in the configuration phase are the following: • Implementation cost • Training costs • Cost of use • Hosting • Technical support • Cost of use per user, depending on the number of users • (Specific developments: integration of other tools, specific functionalities…) As mentioned before, check out if joint-purchases or training moments with other NGOs are possible. This may reduce the financial efforts a little.

Competences In the configuration phase, the selection of the indicators is crucial. The person, who is willing to take this task, must be someone who has a clear understanding of: • • • •

The different processes, ideally across countries Collaboration with IT providers Future users: who to involve in at first? The organizational structure: who reports to whom? Who provides what kind of information?

Often, that person is someone found internally. Tip! Foresee (extra) workforce for the configuration phase. “Continue involving the IT teams, whether they are in-house or external.” User profiles The characteristics above are very important in order to create the different user profiles in the software. Each user profile has specific user rights (role): you can limit views at any level of the hierarchy, from a broad scope (e.g. regionally) or down to specific project(s). Each role determines what the user can and cannot do within the project. A user may have more than one role. Exercise 9: How many and what kind of profiles do you need to create for your organization? What are the specific user rights per profile? To help you determining the different profiles, go back to exercise 3 where you described the different responsible persons at the different monitoring levels. Note that the profiles may also be used for external people, e.g. to give donors insight.

Description of profile Project assistant

Specific user rights (role) 1. This user may only be able to update indicator information but not financial information. 2. This user can compare the indicators of his project to similar projects in other regions. Example

Description of profile Administrator

Specific user rights (role)

For who All the project assistants of the Irrigation Projects

For who

Train the trainer Vendors of EMT software offer trainings to train one trainer inside the organization. Subsequently, the trainer can provide internal training to the rest of the team. â&#x20AC;&#x153;If you organize a training, define your expectations and communicate them with the vendor. What do you want to achieve? Who will participate? We planned a 2 day training with our developer about the different possibilities the software offers in general. This was very useful for the people in our organisation who are responsible for the configuration of the software adapted to our organisation.â&#x20AC;? Tip! Make a concrete planning with the vendor about the content of the training, and possibly use examples related to the specific activities of the NGO. It is quite simple to give them a table with activities and indicators beforehand, so that they can prepare a simulation corresponding to reality. It would allow to reflect immediately on the possibilities and best way to personalize the final configuration.

6. Use Phase (Phase 3) Now the configuration is done, you can start using your new software. Question How to ensure the software is integrated into your daily operation?

Vision Investing in an EMT is a long term investment. Align your monitoring with the long term vision of your organization; not only in the first year, but also in the second year and subsequent years after. Tip! A lot of successful organizations link powerful communication to (interim) results monitored via their monitor system (dashboard). An EMT can therefore also help organizations to respond to the tendency towards more transparency. Example: Cordaid Cordaid believes that sharing information and knowledge will make co-operation easier and thus improve the quality, impact and relevance of our work. The open development movement also aims to transform the development sector itself and the interventions and activities of the actors involved. The organization publishes and updates its projects on a quarterly basis. Cordaid works on the principle that all project information should be published unless a clear reason is provided in accordance to the Exclusion Policy.

Importance During the use phase, ensure feedback is being delivered to and from the initial stakeholders. Is the monitoring software delivering added value? Tip! Monthly â&#x20AC;&#x2DC;topoâ&#x20AC;&#x2122; or clear and concise representation of the state of affairs based on the monitoring dashboard is useful, preferably with color codes and including financial data. Tip! Everyone must interpret indicators in the same way.

Plan Use the following checklist as guidance to plan the use phase. Unlike the preparation and configuration phase, the roadmap for the use phase is an iterative process. The second, third and fourth step will be regularly repeated.

1. Determining leadership

2. Use

3. Analyse

4. Report

5. Evaluation Roadmap for the configuration phase “It is essential to keep one person in charge of global coordination and follow-up at head office level. And it is essential as well to have a relay at field level: at least one motivated staff per program, usually the operational coordinator or a senior project manager, who would be the focal point among his or her colleagues. He or she links with the person in charge at head office level and with the IT support for technical issues.” 1. Determining leadership • Who is responsible for the overall use and commitment? • Who guards the quality? • What if the indicators should be fine-tuned end/or expanded? 2. Use • Who are the users? Is everybody involved? • Who is responsible for internal trainings? • What to do in case of bugs? 3. Analyze • Who manages the overall dashboard? • Who analyses the data? • Are the indicators still relevant? 4. Report • Who does the reporting? • When and how frequently do you need to report? • Which reports are for internal people, which for external people? 5. Evaluation • When do you plan evaluation moments? • Who should be involved? “It is essential for the sustainability of the project to make regular communication on progression, i.e. with testimony from current users, or using it to present the Desk’s activities etc.” Tip! A clear allocation of responsibility to one person per type of data is useful. Tip! Regular use is a precondition to effectively get the true learning lessons from the dashboard. Tip! Having integrated quality checks is very important. It’s important that background information is uploaded and can be verified. 30

Resources Time Every user should foresee specific time in order to enter his/her data. In the beginning this will need some follow-up so that after a while the data entering becomes a habit. Besides that, time is needed for analyzing, interpreting and reporting. What is the frequency you want to report? Exercise 10: Go back to ‘Checklist use phase’ and make for each activity an estimation of the time needed. Financial efforts Depending on the financial model your monitoring software has, your organization may or may not have to pay for specific additional developments. In some open source systems, joint developments are for example paid by public funding. Costs in the use phase are the following: • Cost of use • Hosting • Technical support • Cost of use per user, depending on the number of users • (Specific developments: integration of other tools, specific functionalities…) • Maintenance costs Tip! Everyone in the organization should cooperate.

Competences Train the end user The person, who was trained in the configuration phase by the vendor, should now give internal training(s) to the end users. Tip! An intensive and good internal training is the key to successful implementation. Also foresee refreshment trainings and/or induction sessions for new users. Data processing During the configuration phase, a list was made of all the necessary profiles and their specific user rights (roles). Now we go a step further, and take a look at what specific competences are needed for an excellent use of the software regarding the data processing (analyzing, interpreting, reporting) and finetuning. If the program of your organization changes over time, the need for new indicators could arise. Adding new indicators is not that difficult, because you already have designed your architecture in the configuration phase. 31

Exercise 11: What are the competences you need in the use phase? Are the right persons available in your organization? This exercise will help you to put the distribution of tasks in order. 1. Which competences do you need for the data processing? List the competences in the first column of the table. Consider the following tasks: coordination, contact person with the software company, data analysis, data interpretation, reporting, trouble shooter… 2. Which competences are available within your organization? For which competences do you have to look outside your organization? Check the availability for each competence. 3. For each competence, write down the name of (an) appropriate person(s). 4. What kind of user profile does this person need? Competences needed

Available in your organization? yes / no

Name of appropriate person(s)

Profile

yes / no

“Do not forget or minimize the importance of IT: on one side, infrastructure in the field needs to be sufficiently homogeneous and stable; and a part-time IT resource (in-house or external) is needed to carry this kind of project in a successful way (helpdesk/troubleshooting, maintenance, minor adjustments/adaptation…).”

Tip! Due to technological progress in certain monitoring tools (e.g. offline use), the role of the current person who collects and enters data may shift to more competence in analysis and insight. This may also accelerate the flow of information within the organization. Learning Please remember that you are not alone is this process. A lot of NGOs went through the same process as you, and solved problems they were facing. Make use of the online learning platform from ACODEV and the ngo-federatie to learn from each other, and enjoy learning!

2015