Az Ernst & Young jelentése a Norvég Civil Alap átvilágításáról

atlatszo.hu 2014.06.27.

1. oldal

Draff report - only for discussion

=!l E r n s t &YouNG Quality In Everything We Do 18

atlatszo.hu 2014.06.27.

2. oldal

Ûidft report - only for discussion

Contents Summary findings

20

Conclusions and recommendations

22

Key Findings

23

Selection of sub-projects

23

Project monitoring

31

Financial management, control mechanisms and reporting

35

Attachments

36

Attachment no. 1. List of sample sub-projects

39

Attachment no. 2. Sample projects testing results

41

19

atlatszo.hu 2014.06.27.

3. oldal

Draff report - only tor discussion

Summary findings

Summary of the main findings from the audit of the NGO Fund in Hungary - project no. HU0068 are presented below: At Fund Level The adequacy of the accounting and financial operations and reporting i

^ j ^ ^ J ^ ^ ^ ^ ^ ^ O ' Hungary

The Operator holds separate bank account for the re-granting part of the NGO fund.

Management fee is received on the Operator's general bank account.

Interest accrued on the re-granting bank account was used for further regranting.

%mthet disbufsewiifni-oi ttie r e ^ } f t j & i 0 sub-piojagts;' Hungary

Disbursements to sub-projects are done by the Operator only.

Disbursements are done from a EUR account to a HUF account at bank exchange rate.

Hungary

The Operator has the ultimate responsibility for budget compliance and control.

The budget is controlled on an on-going basis in excel sheets. The budget is broken down to detailed budget lines and costs incurred are input into the sheet and compared against budgeted values.

actual incurred expenditure. Our testing showed that: - the Operator reported its estimated labor costs and actual incurred other direct costs. Overhead costs are allocated to the Fund based on full-time posts allocated to the Fund. The overhead costs reported to the FMO are estimates.

- the Partners (except one who reports actual incurred costs) report fixed fee per budget line. Costs reported by the Partners are not verified by the Operator.

Exchange rates used in guarterly reports are the EC reference rates.

in.placĂŠ te if^etexifetstirS^ftiint^isrojects J r f ^ u t e n ' j M end t&cwmy ef fundi, Hungary

^

Disbursements to sub-projects are tracked by the Operator.

Systems in place for irregularities identification comprise of i.a. on-site visits.

Per information received, there were no cases of recovery of funds until the date of the monitoring.

20

atlatszo.hu 2014.06.27.

4. oldal

Draff report - only for discussion

Hungary

The Operator and its Partners maintain separate bookkeeping accounts for the Project.

Records are maintained in an organized and consistent way.

We noted no issues with internal controls on the Project.

Analysis of the selection criteria S ^ j e c t i o ^ f f ^ ^ a hrisb~prejects,under Hungary

•'• S li

11 L

tfis rtnef ' . s . ' : . : . i . : . : . i . : . : . i . : . : . i . : . : . i . : . : . i

The selection criteria used were compliant with the Implementation Agreement.

The sub-project selection process was transparent with some issues.

n rt i

*i r

i t i in

\["' K

hi

Hungary

Transparent with some issues.

-Ti • . ^ •>

th

a.

t np i th ^

-.

*-

I th

• "it- ]

tu i '*i nn tt,-< f- >r tu

11

11

t | plu t .ri*

- u - t l , i rit m f >i

V111

n

In

*•> ' - >ii » r I j l • or \\ U u K n - i h n Hungary I a

Bulgaria

M-T

The sub-project applicants can complain to the Selection Committee. However, "after the quality evaluation of the applications and approval of projects for .funding by the Selection Committee - the applicants could appeal, but the Selection Committee decisions were final.

-rr. I'Mtk'i th i t i • v. i - ' r i f i t , i n , - . . s K r i t\ ïl \ ipp n lin vtitl u n t i i f f 111 l u iti'.p-, 'Ai i x f I ti". t l v Cr i r t ¿,11 , ir 11 t l . ' " i i,l I n , O P . ï h - t l - r i n . i tl 11 11. u ' . t . -,<\-t, t , \ l v

t h , i trt L. .* -i> -MO\AO , . T I P K -

At sub-project level Adequacy of the accounting and financial operations and procurement reporting systems

Hungary

The beneficiaries receive funds from the Operator on a separate bank account opened tor the re-granting.

Verification of eligibility of expenditures by the Operator and its Partners is not sufficiently documented.

In the sample projects tested - in not all cases 10% of expenses were verified.

Hungary

Accounting records of the beneficiaries are checked during "financial" visits onsite. The Operator and their Partner maintain separate bookkeeping accounts for the NGO Fund.

21

Draff report - only for discussion

BlfcTgSt cerriplbwcff Hungary

*

< < < < < < < < < < < < < < < < < < <

Budget compliance is verified in the beneficiaries" quarterly reports

pjJbfiC.pVQc.ijreWwZoftifeliaht$' * j.

Hungary

atlatszo.hu 2014.06.27.

5. oldal

i»i»s»i»s»s»s»s»i»s»s»s»s»i»s»s»s»s»i»s»s»s»<

The sub-projects' budgets are below the public procurement threshold.

^??^?^5r* ^ j f'?. 1^.^f^y? ^ ^^-*^ i^.T ' ^f r^/f^ y^^j ^ ^^^.^'-*r^^f f^?^ ,T 1 J

Hungary

a

Co-financing and in-kind contribution are verified by the Operator and its Partners.

Interest earned on the sub-projects grant account is deducted from the final grant settlement.

Conclusions

* The NGO has established dedicated team to manage the NGO Fund implementation within the organization, supported by finance/accounting and administrative functions responsible for managing the project. l

-

The NGO established internal procedures supporting processes carried out on the basis of the Fund set-up documents as well as supporting both beneficiaries and other stakeholders in access to relevant information on the project.

* The project is managed in an organized way and it is transparent for the applicants. The project implementation is supported by the IT system designed by the NGO which facilitated applying for the grants and assessments made by evaluators. l

-

The sub-project selection process is transparent and supported by IT tools. At the same time there are some issues related to this process.

* Monitoring of the implementation of sub-projects and reporting of sub-projects is transparent and consistent among the Operator and its Partners. l

-

The present design of the process for verification of eligibility of costs reported by sub-projects does not ensure appropriate audit trail for the checks done.

• ' The management costs reported to FMO consisted of different categories, including direct project management costs and indirect costs, allocated to the project. The allocation of costs to the project, e.g. personnel costs is ca 50% and overhead costs are allocated at 4 3 - 4 7 % of the total costs. Reporting of the management costs to the FMO proved to be inconsistent between the Operator and the Partners.

22

atlatszo.hu 2014.06.27.

6. oldal by the

Draff report - only for discussion

The findings and descriptions of processes in this report are based on information provided representatives of the Fund Intermediary and its Partners involved in the implementation of EEA Grants in Hungary, namely the Hungarian Environmental Partnership foundation ( Ö k o t á r s A l a p í t v á n y ) , Foundation for Development of Democratic Rights (Demokratikus Jogok Fejlesztéséért Alapítvány, Demnet), Autonómia Foundation ( A u t o n ó m i a A l a p í t v á n y ) and Carpathian Foundation (Kárpátok A l a p í t v á n y ) .

The main document defining the EEA Grants management and control system is the Implementation Agreement signed on February 4, 2 0 0 8 between the Financial Mechanism Office and the Hungarian Environmental Partnership Foundation (with annexes and addendums) which lays down the responsibilities of the Fund Intermediary/the Operator and defines the Fund set-up.

We were informed that there were no particular procedures developed for the purpose of EEA Grants implementation in any of the Partners ( Ö k o t á r s , Demnet, A u t o n ó m i a , Kárpátok) apart from the Implementation Agreement and guidelines developed by the FMO and the Operator and its Partners and provided to sub-project applicants and evaluators (described in the following chapters of the r e p o r t ) .

Grant Management Information System (GMIS) In their bid to EEA/Norwegian Financial Mechanism the Operator and its Partners ( O k o t á r s , Demnet, A u t o n o m í a , and Kárpátok) committed themselves to use Grant Management Information System (GMIS). GMIS is a web-based tool (available on www.norvegcivilalap.hu) in which data regarding sub-project applications is recorded, analyzed and processed. The Operator developed an online application form for the potential applicants to fill in and submit online to the Operator and the Partners. The application form could also be printed and submitted in hard copies. We were informed that the online application form contains inbuilt controls, e.g. to check whether the applicant meets the minimum requirements, and the system does not allow for applications to be submitted after deadlines.

GMIS contains i.a. information on when the application was submitted, information about the applicant, evaluators' data, results of evaluators' assessments, an evaluator's conflict of interest, and articles related to the application rounds/calls for proposals, FAO, list of granted projects, premonitoring questions. Access to GMIS is possible for registered users (requires login and password).

All of the above mentioned data is available in the online system only for "admin users" (users appointed from the Operator and the Partners). The "admin users" have access to all data and can input and modify all the data (including evaluators' scores, lists of evaluators who participated in the evaluation, status of projects - rejected/granted). Furthermore, the evaluators have limited access to the system - they can only access the application and their own evaluation sheets. If an evaluator was marked in the system as not independent from an applicant, the system does not allow him/her to access the evaluation sheets for that application.

Selection of external experts to assess the sub-project applications In accordance with Article 4 of the Implementation Agreement: -

"the Operator shall appoint four selection committees,

23

one in each thematic area

(...),

atlatszo.hu 2014.06.27.

7. oldal evaluators

Draff report - only for discussion

-

the selection

committees

shall be made up of a chairperson and four independent

(...), -

the chairpersons Partners

shall be appointed by the Operator;

three of them based on nomination

by the

(...),

-

the evaluators shall be selected based on their expertise and experience in the particular area, as well as their knowledge of the Hungarian NCO sector (...),

thematic

-

the Operator shall invite the relevant ministries selection committee,

on each

-

the evaluators shall be independent from the Operator and the Partners and applicants in their respective thematic area,

-

the selection committee shall operate in an open, transparent and accountable manner, and its composition must ensure that due attention is paid to possible areas of conflict of interest".

to appoint

one of the evaluators

Further, conflict of interest is defined in article 10.4 of the Agreement - "The Operator shall take all necessary measures to prevent or end any situation that could compromise the impartial and objective performance of the Agreement. Such conflict of interests could arise in particular as a result of economic interest, political or national affinity, family or emotional ties, or any other relevant connection or shared interest. Any conflict of interests, which could arise during performance of the Agreement, must be notified in writing to the FMO without delay".

Based on information provided during interviews and testing results on sample projects we noted the following regarding the selection of sub-projects: 2

Selection Committees • There were separate selection committees in each thematic area at each of the Partners.

• Data of the Selection Committee members in each thematic area - the evaluators - are input into the GMIS. Information on any conflict of interest declared by the particular evaluators is also input ("tick mark") in the system and those evaluators cannot assess (and access) those applications in the system where conflict of interest was declared. We were informed that the evaluators did not have access to other evaluators' assessments in the system either.

•The evaluators signed contracts with the Operator and the Partners as well as declarations of impartiality. The declaration of impartiality is either a separate document or a clause included in the contracts in which they declare to obey confidentiality of data and impartiality, which was defined as having no financial interests in the applicants, and neither they nor their close family members work for the applicants. The declarations of impartiality were signed by the evaluators before they knew the applicants and were obliged to inform the Operator or the Partners of any conflict of interest after the names of the applicant organizations were known.

•There were separate trainings for evaluators in each thematic area during which they were informed about the online evaluation system and the assessment criteria.

Okotars - T h e Selection Committee members were chosen from Okotars "pool of evaluators", i.e. persons they were acquainted with and often cooperated with. There was no formal selection process of the independent evaluators.

•There were five persons in the Selection Committee for the three calls in the environmental priority sector: Mora Veronika - the Selection Committee Chairperson (Director of Okotars since 2 0 0 7 ) , Prem Krisztina - independent evaluator, representative of the Ministry of Rural Development,

The sample projects include: 'cut-off projects and 20 sample projects (please refer to Attachment no. 1 and no. 2 for the lists of sample projects).

z

24

atlatszo.hu 2014.06.27.

8. oldal

Draft report - only for discussion

dr Malatinszky Ákos - independent evaluator, Zalatnay László - independent evaluator, dr Kalas G y ö r g y - independent evaluator.

There were also two more persons input in GMIS: Karen Turner - a representative of FMO and László Perneczky - a potential sixth evaluator both of whom eventually did not participate in the evaluations. We were informed that these two persons have "read-only" access to the system and assessments of the particular evaluators.

We were also provided with the CVs of the evaluators and we observed that: dr Kalas G y ö r g y was the president of the Advisory Board of Ökotárs from 1 9 9 1 to 2 0 0 4 , Zalatnay László was the president of Ökotárs from 1 9 9 8 to 2 0 0 2 .

- P e r information received, Ökotárs was not provided by the FMO with a definition "independence" of the evaluators other than the one stated in the Implementation Agreement.

of

- T h e 5 sample projects in the environment priority sector and " c u t - o f f " sample projects were evaluated by at least three evaluators and all of them were evaluated by the Selection Committee Chairperson.

•Ökotárs developed "Rules of valuation" ( " É R T É K E L É S I S Z A B Á L Y Z A T " ) for the evaluators which are accessible online. The Rules describe in general terms how the evaluation process is organized, i.e. the successive steps in the process and deadlines for each step. The Rules were distributed among the evaluators in all priority sectors.

Autonómia • In total there were nine evaluators selected by Autonómia for the three calls in the social priority sector (not all of them participated in all three calls). There was no formal selection process of the independent evaluators - they were selected on the basis of their knowledge of the sector and were acquainted with A u t o n ó m i a .

•We were informed that there was no chairman selected for the Selection Committee in the social priority sector.

•We were provided with CVs of the evaluators and observed that at the time of the evaluation process the following evaluators: - Béres Tibor - was working in A u t o n ó m i a , -

Kelemen Ágnes - during the evaluation phase worked for the Ministry of Social Affairs and Labour and then joined A u t o n ó m i a ,

-

Csongor Anna - was the Director of Autonómia and member of supervisory board of Ökotárs (as well as a member of the managing board of Menhely Foundation - the Foundation applied for grant to Demnet, however was rejected).

•We observed that Autonómia did not systematically use the "tick mark" in GMIS to exclude evaluators with conflict of interest. Per information received, the "tick mark" was used to indicate the number of evaluators in certain rounds/calls for proposals. All of the evaluators had access to applications, however in case of declared conflict of interest they did not evaluate the application.

•The sample 5 projects in the social priority sector were evaluated by three or five evaluators. In case of " c u t - o f f " sample - by two or six evaluators.

Demnet •There were five evaluators in each round/call for proposals in Demnet's priority sector, i.e. civil, but altogether nine evaluators participated due to personal changes during the calls for proposals.

• Per information received, in their evaluator selection Demnet used the following criteria: knowledge and experience in the field of civil organizations' capacity building and university or college degree. Demnet selected evaluators based on their previous contacts and also looked for evaluators from the Ministry of Social Affairs and Labour.

25

Draff report - only for discussion

• We were provided with CVs of the evaluators and observed that: -

Pálvolgyi Péter - was the Managing Director of Demnet since 2 0 0 3 ,

-

Nizak Péter - was the previous Managing Director of Demnet ( 1 9 9 8 - 2 0 0 2 ) ,

-

Jaksa Brigitta - was the Policy Officer of Demnet since 2 0 0 8 ,

-

Garas lldikó - was delegated by the Ministry of Social Affairs and Labour,

-

Eros Barbara - was an employee of Demnet,

-

Csanády Daniel - was Programme Leader in Demnet until 2 0 0 6 .

atlatszo.hu 2014.06.27.

9. oldal

"•The sample 5 projects in the civil priority sector were evaluated by at least two evaluators and maximum five. The " c u t - o f f " sample projects were evaluated by five evaluators.

Kárpátok • In the thematic area of cultural heritage, the applications were separated into three sub-areas: -

intangible cultural heritage,

-

tangible cultural heritage,

-

environmental cultural heritage.

In the selection of evaluators, the Carpathian Foundation sought for evaluators who had basic knowledge and experience in all the above three fields. However, the evaluators were specialized in one of the three fields.

•» The Foundation did not have a "pool of evaluators" to choose from. The Foundation requested the National Office of Cultural Heritage to suggest evaluators. They also acquainted with a few evaluators during the preparation of the bid to FMO. •» We were provided with CVs of the evaluators and observed that during the evaluation process: - Soós Gábor - was the World Heritage International Relations Officer in the National Office of Cultural Heritage, used to be Permanent Delegate to UNESCO, -

Vartik Melinda - was the Financial and administrative leader at Carpathian Foundation, Project Cycle Management trainer, had no experience in the field of cultural heritage,

-

Gyulai Éva dr. - was a historian and museologist,

-

Koles Sándor - was the Executive Director of the Carpathian Foundation in 1 9 9 6 - 2 0 0 5 ,

-

Déme Péter - was delegated by the National Office of Cultural Heritage.

-

Hortiné dr. Bathó Edit - was a musem director.

• In every call for proposals there were five evaluators. And in the 5 sample projects in the cultural priority sector and " c u t - o f f " projects were evaluated by five evaluators.

Sub-project Selection process - O p e n calls for proposals that were published on the program website (www.norveocivilalap.hu) contained detailed information on the eligibility of applicants, the application procedure, list of non-eligible costs and non-eligible activities, the general and detailed selection criteria, eligible thematic areas and eligible activities, description of contracting and reporting for successful applicants. However, there was no information on scoring of particular criteria or the weight given to the criteria.

- T h e applicants and stakeholders have access to relevant information on the project and selection processes on the project website www.norveocivilalap.hu as well as the Partners' websites.

•The third call for proposals was different from the first two (one stage) calls. Based on the evaluation of the first and second call the Operator considered that the quality of projects' proposals was lagging, i.e. in the Operator's opinion there were many quality projects but they could be better prepared and better defined in the sub-project applications. Therefore, two-stage

26

atlatszo.hu 2014.06.27.

10. oldal more effectively. In

Draff report - only for discussion

procedure was proposed so that the applicants could use the Operator's help the first stage the applicants were requested to provide a brief project idea which was evaluated and selected projects were then allowed to the second stage. In the second stage the projects were evaluated based on the detailed selection criteria in accordance with the Implementation Agreement. After the first stage the Operator carried out workshops for the applicants on how to develop a project.

â&#x20AC;˘ After the applications were submitted in the system, the evaluators were assigned applications to evaluate. The evaluators input their assessments/score into the system on-line and could modify their assessments until they submitted the final score in the system. GMIS contains predefined criteria to be scored (in accordance with criteria cited in the open calls) and scores to be selected for each criterion from a drop-down list. The evaluators could also write their comments regarding the applications. The system calculates average score per project and ranks the projects according to their scores. The preliminary ranking lists were generated from the system and provided for the Selection Committee meetings.

For example in case of the l

b t

call the criteria were defined as below:

How relevant is the proposal to the objectives and priority sectors of the NGO fund?

How relevant is the proposal to the objectives of the thematic area?

How clearly defined are the goals and targets of the proposal?

How relevant is the proposal to the needs and constrains of the target group?

To what extent are the target group or other stakeholders/constituencies involved in the project?

To what extent does the project contain innovative elements?

How coherent, achievable and appropriate are the proposed activities?

To what extent is the proposal built upon real partnership and cooperation with other organizations/institutions?

To what extent does the proposal contain objectively verifiable indicators for measuring project results/outcomes?

To what extent is the budget clear and detailed? To what extent is the proposed expenditure necessary for the implementation of the project?

To what extent does the proposal contribute to the strengthening and sustainability of the applicant and of the NGO sector?

Will the capacities and experience of the applicant and its partners allow the successful implementation of the project both in technical and financial terms; and to achieve the expected results?

Maximum total score

0 - 15 0 - 10 0 - 10 0-10 0 - 15 0 - 10 Not used in scoring, thought as discriminatory 0-5 0-10 0-5 0 - 10

100

In the second and the third call the scoring was different. Maximum score for each category (there were three major criteria: objectives, activities, budget) was 5 points and the scores from each category were multiplied, e.g. 5x5x5, hence the maximum total score was 125. Then average score of all the evaluators was calculated.

We were provided with the guidelines for evaluators for the third call and noted a discrepancy the guidelines state that the scores will be added not multiplied, i.e. the maximum score is 15 not 125. â&#x20AC;˘ The GMIS shows composition of the Selection Committee (evaluators) for each project, as well as their scores and comments, however comments made by evaluators were usually very brief and not consistent - some of the evaluators wrote no comments at all to justify their scores.

5kotars - T h e sub-project applications were discussed during the Selection Committee meetings. Per information received, the Selection Committee's goal was to exclude applications that did not have a chance to succeed (e.g. in the first call there were 154 applications and 60 were discussed during the meeting). Ranking lists of all projects generated from the system (with identification numbers of applications, titles of projects, NGO's name, scores given by each evaluator and

27

atlatszo.hu 2014.06.27.

11. oldal third call)

Draft report - only for discussion

average score in case of the first call, and only average score in case of second and were provided for the Selection Committee meetings. The evaluators were asked to propose projects for discussion, i.e. propose projects that should be accepted. The evaluators also proposed where the " c u t - o f f " line for projects accepted and rejected for funding should be.

We were provided with meeting minutes from the Selection Committee meetings. The minutes contain brief information on which evaluator proposed to discuss which project and the final list of projects accepted. The minutes do not contain justifications for projects to be accepted or rejected (e.g. for projects that were scored below the ' c u t - o f f line and were accepted in the final version of scores - there is no written justification for the decision). The final scores agreed upon by the Selection Committee were recorded in GMIS (historical data is stored in the system).

• In our sample of 5 sub-projects and "cut-off" projects in the environmental priority sector there were cases where score of a project was changed, i.e. increased and there was no written justification for the change provided - neither in the Selection Committee meeting minutes nor in the GMIS (refer to Attachment no. 2 for details).

Autonómia • In the case of Autonómia the ratio of applications and granted projects was the following: First call for proposal: 19 projects were granted out of 2 4 6 applications, Second call for proposal: 2 0 projects were granted out of 3 4 1 applications, Third call for proposal: 15 projects were granted out of 3 1 2 applications ( 4 9 applications were evaluated in the second phase).

• In the first and second call about 2 0 0 sub-project applications were received. Autonómia divided the applications into two groups and there were two evaluation teams established with 2 or 3 evaluators.

Each evaluation team selected the best applications ("their favorites") to be shortlisted. Afterwards, all shortlisted applications were read through by all evaluators.

The evaluators decided about the application's rejection or acceptance on the Selection Committee meeting and the final scores were defined during the meeting and input in GMIS. • In our sample of 5 sub-projects in the social priority sector and "cut-off" projects there were cases where score of a project was changed, i.e. increased and there was no written justification for the change provided - neither in the Selection Committee meeting minutes nor in the GMIS (refer to Attachment no. 2 for details). We were also not provided with details of the preliminary scores awarded by each of the evaluators - only average scores per project were available and the final scores input in GMIS. Demnet • In the case of Demnet the ratio of applications and granted projects were the following: First call for proposals: 2 4 were granted out of 3 5 2 applications, Second call for proposals: 19 were granted out of 187 applications, Third call for proposals: 12 were granted out of 185 applications.

- I n the first call, applications were evaluated on average by three evaluators and there were separate Selection Committee meetings for macro and micro sub-projects.

•In the second and third call there was a pre-evaluation by evaluator couples and the best 4 0 applications were then evaluated by every evaluator. The applications were evaluated individually, and the evaluators learnt about other evaluator's assessment only during the Selection Committee meeting. The evaluators made their individual rankings of applications and the rankings were summarized during the Selection Committee meetings. The applications were then discussed.

• The original evaluation scores (before the Selection Committee meetings) were input in the GMIS. The scores modified after discussions during the Selection Committee meeting were recorded in

28

atlatszo.hu 2014.06.27.

12. oldal were accepted for funding and some

Draff report - only for discussion

Excel sheets. We noted that some projects with lower scores projects with higher scores were rejected and the reasons for the decisions were not documented (refer to Attachment no. 2 for details on results of our t e s t i n g ) .

Kárpátok

•» In the case of Kárpátok the ratio of applications and granted projects were the following: First call for proposal: 1 1 macro projects and 11 micro projects were granted out of 2 0 1 applications, Second call for proposal: 2 0 projects were granted out of 2 3 0 applications, Third call for proposal: 2 0 projects were granted out of 2 6 0 - 2 7 0 applications ( 1 9 signed contracts out of 2 0 ) .

•» In the first call all applications were evaluated by five evaluators.

• In the second call Kárpátok formed evaluators' teams - one with 2 evaluators and the other with 3 evaluators. The evaluator teams evaluated the applications, ranked the applications by scores and those applications which received more than 1 point were suggested for further evaluation. These suggested projects were then evaluated by all the 5 evaluators.

(In the second round the applications were evaluated in three aspects, and each aspect could be scored 0 - 5 . The aspects: 1) project goal 2 ) activity 3 ) budget.)

• In the third call there were two selection rounds. In the first round 3 evaluators participated, and each evaluated each project idea. The selected project ideas were then evaluated by all the 5 evaluators.

• In the first and second call not all the available funding was distributed. In accordance with the information received, the goal was not to distribute all the funding, but to finance those projects which in the Kárpátok's opinion were worth implementing. The funding not contracted in each round could be used in the next call - the Partner did not lose the funding. In the third call all the funding was granted to the applicants.

"We were provided only with ranking list of projects with their final scores (i.e. scores stored in GMIS). There was no information available on whether the scores were changed during the Selection Committee meeting.

Complaints • Per information received and the open calls we noted that the Selection Committee decisions were definite and the applicants could not appeal: "Applicants rejected due to non-compliance with eligibility (administrative) criteria can request the Operator to review its decision within 10 days of being notified about the rejection. After the applications are evaluated based on eligibility criteria the selection committee's decision is definitive and the applicants have no right of appeal at this stage."

- W e were informed that the applicants could request more information from the Selection Committee but their decision would not be changed regardless the applicant's explanations.

• Per information received, there were very few complaints by the applicants and rejection due to formal requirements occurred only occasionally thanks to the effectiveness of the online system.

The cause of formal rejection was in most cases that the legal entity type of the applicant was different than the required/eligible types in the call for proposal.

• We selected sample projects to test the complaints mechanisms. The results of our test show that:

29

atlatszo.hu 2014.06.27.

13. oldal

Draff report - only for discussion

Ökotárs

In the sample projects rejected after the quality assessment ( N C T A - 2 0 0 9 - Z 0 4 7 1 5 , NCTA-2008Z 0 2 1 6 0 , NCTA-2009-Z03821) there was an exchange of letters. Ökotárs did not change their decision.

Autonómia Per information received there was only one complaint submitted Autonómia did not change their decision.

(NCTA-2009/2-S06624).

The official explanations for rejecting or accepting projects can be found in the online system.

Otherwise, A u t o n ó m i a did not document the selection process.

Demnet In the sample projects rejected (NCTA-2009-C04764, NCTA-2009-C03775, NCTA-2008C 0 3 0 5 3 ) two projects were rejected due to administrative criteria, one of which was later allowed for evaluation based on the applicant answer. In case of the third project which was rejected, the applicant inquired about evaluation criteria, scoring.

Kárpátok In the sample projects rejected ( N C T A - 2 0 0 8 - K 0 2 1 9 0 , NCTA-2008-K01966, NCTA-2009K 0 4 7 0 5 ) the applicants inquired about detailed reasons, scoring, and how to prepare for the next call.

Findings/conclusions The project selection process is transparent and supported by IT tools. At the same time there are some issues related to this process: • Procedures applied do not ensure that the risk of conflict of interest is effectively managed. Based on our testing some evaluators were not independent from the Operator or the Partners at the time of the application appraisal.

- T h e r e was no formal selection process for the independent evaluators and the selection of evaluators was not documented/justified in written. In our testing, one of the evaluators had no relevant experience to evaluate the applications.

•The applicants were not provided with the entire information on the evaluation of applications the criteria were listed in the calls for proposals however the maximum scores for particular criteria were not provided.

•The project selection process was based primarily on evaluators' assessments of projects.

Evaluators assess both content related aspects as well as cost effectiveness of the project. Based on a sample of project documentation reviewed by us, we noted that quality of justification was not uniform in case of particular evaluators (some gave comments, some did n o t ) , and there is no written justification provided for the final scores (or changes of scores) after the Selection Committee meetings.

•Assessment of budget aspects of the project is an integral part of evaluation. However it is questionable if external evaluators were suitably equipped to carry out this assessment. They did not have data on standard costs or historical data from other applications which would be useful for comparisons. They only relied on their individual, varied knowledge and experience about costs of projects. In case of Kárpátok there were no changes to sub-project budgets proposed at all.

•The sub-project applications were evaluated by different number of evaluators ranging from two to six.

30

atlatszo.hu 2014.06.27.

14. oldal different data

Draff report - only for discussion

- T h e sub-project selection process is not uniform among the Partners. There are stored in GMIS - three of the Partners input final scores in GMIS, one - the preliminary scores, and the "tick marks" are not consistently used by all the Partners.

- The chairperson was also evaluating the applications and in the sample projects there were cases where she changed her scores which in turn influenced the applications' chances of receiving the grant. The applications were granted the funds based on their average scores. The transparency of the selection process would be enhanced if there were written justifications for the change of the scores.

- T h e "admin users" of GMIS until the date of the monitoring could modify data regarding the applications and their appraisal in the system. The data stored in GMIS should be uniform in all the Partners and should be locked for modifications after the selection process was finished.

-It is not stated in the implementation Agreement that the chairman/delegated by the Operator should not evaluate applications or whether only independent evaluators should assess applications.

Voting/non-voting members of the Selection Committee were not defined in the Agreement.

- The Implementation application.

Agreement

does not state the minimum number of evaluators

to evaluate an

- The definition of independence should be prĂŠcised in the implementation Agreement, e.g. conditions regarding working relationship could be added - for instance 'did not work in the applicant organization in the last three years before the call for proposals".

Project implementation monitoring Article 5 of the Implementation Agreement defines the Operator's responsibilities regarding monitoring of sub-project implementation. These include: "The end-recipients

shall submit quarterly reports and a completion report

The operator shall further define and implement a monitoring gives the Operator sufficient information on the activities in the

(...),

and reporting sub-projects,

structure

which

For macro grants, the Operator shall conduct on-site auditing visits of each end-recipient to verify proper use of funding. The auditor will randomly select and check at least 10 % of all receipts, For micro grants, recipients (...)

the Operator

shall conduct on-site auditing

visits of at least 10% of end-

The Operator shall conduct desk auditing for all other end-recipients. Each end-recipient will be reguested to present proofs of particular expenses selected randomly by the Operator, in order to audit 10% of all expenses of each end-recipient, if any discrepancy is found, a complete financial audit shall be carried out".

Based on information received and testing of sample projects we noted the following: 3

- The sub-projects were provided with guidelines on reporting and standard forms of reports.

- T h e sub-projects in each priority sector submit quarterly reports to the responsible Partners. The reports are standard form reports in which the sub-projects provide information on financial progress - payments made (the report contains details of financial/accounting documents in each budget l i n e ) . The reports are sent by email and also contain narrative about physical progress.

3

20 sample projects (please refer to Attachment no. 1 for the list of sample projects).

31

Draft report - only for discussion

The quarterly reports are reviewed by the respective Partners and the Operator.

atlatszo.hu 2014.06.27.

15. oldal

Exchange rates. The financial progress is reported by the sub-projects both in HUF and EUR - the sub-projects use the exchange rates that were applied by the bank in the installment transfers from the Operator (the Operator transfers the advance payments from a EUR account to a HUF account of the sub-project).

• The Partners and the Operator also use Interim and final report evaluation sheets of sub-projects which contain information on: -

financial data (budgeted, reported, approved),

-

list of checked documents,

-

evaluation of the implementation of the project comparing to the originally planned/modified goals,

-

main activities, indicators and results; evaluation of the financial implementation of the project comparing to the originally planned/modified budget;

-

recommendations on the approval/rejection of the interim/final report and the amount of final payment/payback.

•The Operator aggregates information about submitted reports in one file. The file serves as a "database" of sub-projects, where information about the beneficiary and the project is stored. The file also contains data on when the reports were approved, the deadlines for the following reports, payments made, dates of monitoring visits, amendments to grant contract. The file is part of the Operator's quarterly reporting to FMO. •The Partners compare the total payments by activity to the total budget in the sub-projects quarterly reports (which are cumulated in each q u a r t e r ) . These are done in electronic form only.

•The Partners also perform on-the-spot visits and the visits are documented on standard form reports. Ökotárs Most often the visits are carried out by Program Manager (Ökotárs Director and one more employee of Ökotárs also sometimes participate in on-the-spot visits) and standard form reports (Site Visit Report Sheet) are issued from the visits. The reports contain information regarding the current status and progress of the project, main findings and impressions (of the Program Manager), status of the use of the grant, issues and possible problems related to the budget and whether the expenditures related to the NGO Fund grant are handled separately in the accounts and list of documents checked and related findings.

We were not provided with procedures/guidelines regarding on-the-spot controls. We were informed that the Ökotárs personnel discussed together what should be verified on-the-spot.

Autonómia Program manager and project assistants carried out monitoring visits regularly. Each project was visited at least once.

Demnet Program leader and project assistants visited every macro project at least once. The micro projects were only visited when the visit was considered by Demnet as reasonable and necessary.

Demnet visited project events and there were monitoring visits as well, when the physical progress and financial implementation of the project was controlled, and sample invoices were checked onthe-spot as well.

Kárpátok Program leader and project assistant visited every macro project 1 or 2 times. More than 9 0 % of the micro projects were visited on-the-spot.

32

atlatszo.hu 2014.06.27.

16. oldal

Draft report - only for discussion

Eligibility of expenditures

In accordance with Article 3 of the Implementation Agreement - Description of sub-projects - the eligibility of costs is defined as "costs are eligible if they are justified, necessary and appropriate, and directly related to the sub-project's long term effects and planned results. (...) The Operator shall develop a more detailed description of eligible costs and make it available on its website no later than at the opening of the first call."

During the audit we were not provided with procedures/guidelines regarding the verification of eligibility of costs by the Operator or the Partners.

The sub-projects were provided with guidelines on what were the eligible costs and what source documents were sufficient to confirm the costs were incurred (compliant with FMO guidelines).

Based on interviews and testing of sample projects' we observed the following regarding eligibility of expenditures: 1

• Per information received, the Operator submits to FMO quarterly reporting packages comprising of a narrative and Excel sheets with projected cash flows, management costs and sub-project progress status. PIRs are prepared by FMO based on the quarterly reports.

•We verified procedures performed by the Operator and the Partners based on the reports submitted by the sub-projects, i.e. we verified whether the Partners checked at least 10% of all expenses in the sample projects and we selected sample expense items for our testing of eligibility of sub-project expenditures and re-performed expenditure eligibility check.

•The Partners and the Operator select expenses to be checked from all the costs reported by the sub-projects in the quarterly reports and request copies of financial/accounting documents supporting the selected costs.

•The Partners and the Operator prepare Excel files with the selected expenses and perform their verifications in the files.

• The results of our testing showed: Ökotárs Per information received, Ökotárs at the end of each sub-project implementation selects at least 10% of invoices/cost items from all the costs reported by the sub-project in quarterly reports and requests copies of financial/accounting documents supporting the selected costs.

We were informed that Ökotárs prepares an Excel file with the selected cost items and marks verified items with "OK" or with a short comment. The checks are performed by the Program Manager with help of interns. We were not provided with any written checklists or alike that would confirm the verifications performed.

Results of testing of the 5 sample projects in the environment priority sector show that: -

Ökotárs requested copies of at least 10% of invoices/cost items from all the items reported by the sub-projects in quarterly reports with exception of one case where they checked 7% of invoices.

-

For each of the projects, an Excel sheet with the selected documents was prepared.

-

The documents provided by the sub-projects were uniform and we noted no issues with eligibility of the items selected for our testing.

Autonómia In our sample of 5 projects in the social priority sector, in three cases less than 10% of expenses were checked.

20 sample projects (please refer to Attachment no. 1 for list of sample projects).

33

Draft report - only for discussion

Per information received, Autonómia used the following selection criteria for expenditures -

at least 10% of the invoices,

-

invoices with high values,

-

at least one invoice from each budget line,

-

at least one invoice from each quarter,

-

own contribution (in kind and in cash as w e l l ) .

atlatszo.hu 2014.06.27.

17. oldal testing:

In the case of sub-projects where less than 10% of expenses were checked Autonómia explained that in the projects there were numerous invoices of low value and Autonómia focused on invoices with higher values. We were informed that no further checks were planned to achieve the 10% of expenses checked.

The verifications were not documented and Autonómia hired students to check the eligibility of expenditures.

We noted no issues with eligibility of the items selected for our testing.

Demnet In our sample in each case at least 10% of invoices were checked. We were informed that in case of problematic projects all invoices were checked.

Per information received, Demnet used the following selection criteria: -

at least one invoice from each budget line,

-

invoices with high amounts,

-

own contribution (in kind and in cash as w e l l ) ,

-

invoices paid after eligibility period,

-

in-kind contribution (checked there were no cash transfer, whether volunteer work was supported with volunteer agreement, timesheet, reasonable f e e ) .

The verifications performed were not documented.

We noted no issues with eligibility of the items selected for our testing.

Kárpátok In our sample in each case at least 10% of invoices were checked and in the case of problematic projects they checked all invoices.

Per information received, Kárpátok used the following selection criteria: -

at least one invoice from each budget line,

-

at least one invoice from each quarter,

-

invoices with high amounts,

-

own contribution (in nature and in cash as w e l l ) .

The verifications were not documented.

We noted no issues with eligibility of the items selected for our testing.

Findings/conclusions • Monitoring of the implementation of sub-projects and reporting of sub-projects to Fund intermediaries is transparent and consistent among the Partners and the Operator. The monitoring visits conducted are compliant with the Implementation Agreement, i.e. the required % of projects to be visited was achieved.

•There are no guidelines/procedures for performing on-the-spot visits. Standard reports are used by all the Partners.

34

Draff report - only for discussion

<• There are no guidelines/procedures for verification of eligibility of costs and the present the verification process does not ensure appropriate audit trail for the checks done.

atlatszo.hu 2014.06.27.

18. oldal design of

• In the sample projects not in all cases 10% of all expenses was verified by the particular Partners.

• We found no issues in the sample projects regarding testing of eligibility of expenditures.

• It was not defined in the Implementation Agreement whether the audit of 10% of all expenses of each end-recipient regarded the number of expenditure items or their value.

In accordance with the Implementation Agreement the Operator "shall provide quarterly reports" and "submit its invoice with the quarterly reports".

the FMO

with

In accordance with Article 8 of the Implementation Agreement "the Operator shall keep full accurate and systematic records and accounts (...) Such records must be kept for a 5-year period after the final payment made under this Agreement.

These documents comprise any documentation concerning income and expenditure and any inventory, necessary for the checking of supporting documents".

Based on interviews and testing of sample projects we observed the following: 5

•The Partners and the Operator maintain separate accounting for the EEA Grants project and separate bank accounts for the EEA Grants re-granting.

Public procurement •We were informed that there were no public procurement proceedings in any of the sub-projects managed by the Partners and the Operator, i.e. in sub-projects in all priority sectors.

•Per information received, the threshold above which public procurement is required by national law is equal to 8 0 0 0 0 0 0 HUF or 3 0 0 0 0 EUR. The highest amount of re-granting that could be awarded to applicants was 80 0 0 0 EUR. Per information received and according to the budgets of the sample sub-projects tested, the single budget lines were lower than the 3 0 0 0 0 EUR threshold, except for personnel costs.

Reporting to FMO • Okotárs submits to FMO quarterly reports which comprise of a narrative of activities performed in a given quarter, management costs of all Partners and information on sub-projects.

• Okotárs keeps accounting records and separate accounts for the EEA Grants project's direct costs.

Overhead costs are recorded on general accounts and then allocated to particular projects using the workload of personnel, i.e. % of work done for the particular project in the total workload.

• Okotárs' financial statements are audited by external auditors annually. We were informed that Okotárs is not required to have regular external audits by law.

•We verified that Okotárs transfers payments to its Partners (management fees) and all the subprojects (re-granting) and we also verified on the sample of 2 0 sub-projects that the exchange rates quoted in Okotárs quarterly reports on the status of sub-project implementation sent to FMO were the exchange rates stated on bank statements (the same exchange rates were used by the sub-projects in reporting to the Operator and the Partners respectively, their project costs in EUR).

5

20 sample projects (please refer to Attachment no. 1 for the list of sample projects).

35

Draft report - only for discussion

• For Okotárs we tested the quarterly reports from June 2 0 0 8 to November 2 0 1 0 and that:

atlatszo.hu 2014.06.27.

19. oldal observed

-

Reports with management costs are prepared by the Okotárs Director. We were informed that data regarding the Partner's management costs is filled in by the respective Partners and is not verified by Okotárs.

-

Management costs of Okotárs: Exchange rates. We verified that exchange rates used in the quarterly reports (June 2 0 0 8 to November 2 0 1 0 ) are the reference monthly rates published by the European Commission.

Personnel costs (without expert f e e s ) . We verified personnel costs reported in October 2 0 1 0 in detail (i.e. reviewed source documents) and noted that in case of Programme director, Financial manager and Programme assistant ca 50% of the employer's total salary cost is allocated to the project. We were informed that this allocation is based on % of a person's total work-time spent on EEA Grants project. However, Okotárs does not maintain e.g. timesheets with time spent on different activities/projects. Moreover, the personnel costs reported in all guarterly reports for 2 0 0 8 , 2 0 0 9 and 2 0 1 0 do not reconcile with the costs actually incurred.

We were informed that Okotárs prepared estimation of personnel costs for each year and divided it by number of months to arrive at average monthly personnel cost which was then input in the guarterly reports. We reviewed the estimates prepared by Okotárs and in case of 2 0 0 8 and 2 0 1 0 the estimates and amounts quoted in the quarterly reports were different, e.g. quoted amounts are higher than the estimated in case of Programme coordinator.

Other direct costs. We verified direct costs of the EEA Grants project (i.a. travel costs, website maintenance) and reconciled to source documents. We noted no issues.

Overhead costs. We were informed that overhead costs are allocated to the EEA Grants project based on number of full-time posts (i.e. full-time posts total allocated to EEA Grants project compared to total full-time posts - ca 4 3 - 4 7 % depending on y e a r ) . Per information received, the overhead costs reported to FMO were the estimated overhead costs allocated to EEA Grants at the beginning of the year did not exceed the budget. We were informed that the overhead costs were underestimated at the time of budget preparation.

Autonomía Per information grant contract. and if they fulfill documents exist

received, Okotárs and Autonomía signed a 'contract of services' and not a According to Autonomy's opinion they accepted to carry out certain tasks the tasks they are entitled for the whole fee. Therefore no detailed supporting for the management costs (we verified May and November 2 0 1 0 in d e t a i l ) .

Demnet Dement reports fixed fee in the quarterly reports for management costs, i.e. fees in EUR agreed in the project budget. Then the fees in EUR are calculated into HUF using the European Commission monthly reference exchange rates (from December 2 0 0 8 to November 2 0 1 0 , in June - November 2 0 0 8 exchange rate of 2 5 1 , 4 3 HUF was used, for which we were not provided with an explanation). Hence, the amounts in HUF do not reconcile to source documents (we verified in detail management costs in September-November 2 0 1 0 ) .

Kárpátok We verified management costs for October 2 0 0 9 management fee were supported by source documents.

36

in detail.

Expenditures

relating

to

atlatszo.hu 2014.06.27.

20. oldal

Draft report - only for discussion

Findings/conclusions

•The project is realized according to plan both from financial and technical perspective. The Operator and the Partners established dedicated teams within the organizations, supported by finance, accounting and administrative functions responsible for managing the project.

•Operations of EEA Grant project are recorded on separate bookkeeping accounts by all the Partners and the Operator.

•Exchange rates quoted in the quarterly reports submitted to FMO by the Operator were appropriate as well as the exchange rates used by the sample sub-projects in reports to the Operator and the Partners.

•Management costs reported in the quarterly reports in all but one Partner are not the actual incurred costs. Okotárs reported estimates and its Partners reported fixed fee in EUR which was converted to HUF in the reports.

•The personnel costs in the Operator are allocated to the NGO Fund project based on the Operator's director judgment - the Operator does not maintain timesheets ( i t is a small organization and timesheets are considered unnecessary by the Operator).

•In accordance with the Implementation Agreement management fixed fee was defined with the Operator. Reporting actual costs on a guarterly basis seems unnecessary. FMO could verify management costs of the Fund intermediaries at bidding phase.

37

atlatszo.hu 2014.06.27.

21. oldal

Draft report - only for discussion

Attachments

38

atlatszo.hu 2014.06.27.

22. oldal

Draft report - only for discussion

Attachment no, 1 . List of sample sub-projects List of sample sub-projects (as at end of January 2 0 1 1 ) . c OJ E c o

l_

> o o "if ITCO

•in E '

O IM

fr

03

.

.VI

^~

s

L

-C-

o

(Ü

as r-j d O IM

CM

d

d

d

*—i *—i o CM

•—i o rj

O

O

O

o

o

o

m

r~-

—

•f 'S r-j

vi £

E •ai ^

^ a; &

< <= O Q.

CO

o o

a a)

to —

•5

>ra

CD

CG SO ai TJ -i—i •ai tz •OJ

CO

>.v-

o

Z

N

f)

I — IM <J o

< ri— m u o 2

m IN

>

i_

N

?5 <

CO

I— "if u o Z

>

OJ

TU VI :ZJ '<U VI OJ > aj l — UJ M

CD 13

"J

O

E

M

o o

< <J

Jj •zi i/i Qj >• CP LÜ

O

o o

O

a t> in

o "if

O

o d

o o

CM

CM

co

0> o

IM

u

in

*

o

o <

O

CM

CO N

CO

o

03

t/i

C

o o in

O O

*—i

CO

•it

-Ol

ci

o

in co

in •—

O

a

TO ID

d

O

pes

o

-Me

111-

•—1

c

•CO

™ to •i= ^c

>

OJ

a. < co

-

o

CT"°

M

—

- s

>

•CO CO

•a

3 — ™ i n </i n) _ a) — c >•

CO CO

Q

a

CO

CO

< o> i — '—i <_> o

<

CJl

<

CJl

2

U

2

U

1

ca

u l/j co •co

UJ

o

o

in o

-Z- <J>

N

39

I— u

o o

i— m u a

U

CO

0>

i - \0 u o ^ u o

C

CO "CO S= > ^ ro CL

< <

atlatszo.hu 2014.06.27.

23. oldal

Draft report - only for discussion

o co

o

o

o

CO

CO

CO

o

o

yO

CO

o o r~-" m o>

O CO

03

m

CO

3

3 (J

3 (J

3

o o

o o

O

o CM

o o

o

CM

CO 0>

o> o>

d

d

CO

CO

O 1X1

O

y3

o

o

o

o

CM

CM

CM

CM

CM"

CO

CM

CM

yD

d

o o

o

o

o

CM

CM

CM

'—1

'—1 '—1 d

CO CM

CM

o d

O

'—1 d

o

O

o

»—i CM

CO

o

CM

»—i

CM

CM

O

o

»—i

o

CM

o

CO

CM

•—1 •—1

CM

o

CM

•—1

d

d

o

o

O

o

o

CM

CM

CM

O

ro

CM 0> O

d

CM

»—i

CO

d

»—i

o

d

o

d

o

o

o

o

o

o

•—i CM

CM

o

O CO 0>

CO

o d

»—i

o

o

o

»—i

»—i

d

d

o

o

CM

CM

CM

o o

o o

o o

d

Oi

d

o o o

Oi CO CO

03

o

o

CO

CO

o

d

o

o

o

o

o

o

o

CM

CM

CM

o o

o o

O

»—i

CM

CM

CM

O

O

o

o o

03

o o

o

o d

»—i LO

o yD

c

4-i '—

CO

'a. _ca

c

< >

CP

9

•CO

s-

£

fa

aj

un

i_ ai

c Û_ "fO

Q

I ¬ ai c fa

•OJ : 0

CO

¥L ¥L

IE 03

.

I—

O

2

<

t/i OJ

•aj 'aj j i t (Z •aJ •CO

IN

c/3

^

ai

C/1

_i:

CO

•—

U"l

>-

CP LjJ CP CP •CO ul ^ra M O _c CO m JO > * L_ CP O fa O. Ul

izr

o

>-

E >

LU ,4-4 Ci. SO CO

jit

aJ IN

<

t/i •OJ CO c

>

Q

03

O O

u

^

>

—

-OJ

CO

•OJ L_

>' CP

o co

< I—

u 2

CM

:0

03 CM

£

™

i£ .£ •o ' a .

o co

0>

ro u o 2 co

< I—

CJi 03 O LO

u

o

o

2

CM CO

< o> un i — o io u o o 2

CM CO

u yD

40

o

CP

•CO

> o

t/l

<

4-4 1=

a

S

M 1/1

«? >N

:0

< I—

u

OJ

I Si. 1 ?

co-g

CO

CP J = CO - a j

O

OJ

ai

o

o

I—

o

>•

IN

:0

03

O

°s

Oi

CJi

•OJ CP •aj t/i

4-4

ZS

—

ul

ér

•CO

aj

Jo

>

"g

o

•—i

o

CO

-o

CO

d

o

CO

o

CO

d

o

yD

CM CM

CM

o

»—i

r-"

CM

CM

o

d

•—i

ai CM

o

?3 < o

<

u

u

I—

o

h Z

c j i un o in o o N

^

Draft report - only for discussion

Attachment no. 2. Sample projects testing results

atlatszo.hu 2014.06.27.

24. oldal

Okotars "Cut-off" sample results. The minimum score for project to be accepted was 74.

ïMthî

First round -' Project

l

2

Project no. NCTA-2008Z00562

NCTA-2008Z02399

WÊÊÊÊÊ

pYicltiiris

3

1

2

NCTA-2008Z01991

NCTA-2008Z01884

3

NCTA-2008Z01699

NCTA-2008Z02355

Preliminary scores Average score received

71,50

73,00

73,75

74,20

75,40

75,59

Number of evaluators

4

5

4

5

5

5

Evaluator's highest score

84

84

81

87

86

88

Evaluator's lowest score

41

64

55

51

34

53

71,50

73,00

73,75

78,59

79,59

78,59

V.Mora changed scores: 51 to 73 the reason of changes is not documented

V.Mora changed scores: 34 to 55 the reason of changes is not documented

V.Mora changed scores: 68 to 79 L.Zalatnay changed scores: 53 to 57 the reason of changes is not documented

Final scores Average score approved Reason for change of N/A scores (if applicable)

N/A

N/A

• In case of sample sub-project testing ( 5 in the environmental s e c t o r ) , our results show:

sub-project no. NCTA-2008Z01081

NCTA-2008Z02323

NCTA-2009Z03705

NCTA-2009Z04847

NCTA-2009/2Z06495

Preliminary scores Minimum score to be accepted

74

74

50

50

43

Average score received

79,66

78,59

46,8

82,4

45

Average score approved

79,66

78,59

66,59*

82,4

45

Number of evaluators

3

5

5

5

5

Evaluator's highest score

95

91

125

125

80

Evaluator's lowest score

51

64

12

27

9

Final scores

"The changes were not documented and there was no information available during audit on the particular evaluator's scores - it is not known who changed their scores.

41

atlatszo.hu 2014.06.27.

25. oldal

Draft report - only for discussion

Autonómia » The results of " c u t - o f f " testing show (minimum score to be accepted - 6 1 ) :

project project no.

1 2 3 1 NCTA-2009- NCTA-2009- NCTA-20O9- NCTA-20O9S04291 S04654 S03375 S04204 preliminary scores average score N/A not 56,6 58,16 61,33 received shortlisted number of No No No No evaluators information information information information available available available available evaluator's No No No No highest score information information information information available available available available evaluator's No No No No lowest score information information information information available available available available comments by No No No No evaluators information information information information available available available available final scores average score 56 56,6 58,16 61,33 approved number of 2 6 6 6 evaluators evaluator's 64 125 125 125 highest score evaluator's 48 1 8 1 lowest score Per information received, the score was changed after monitoring visit at the 1

2 NCTA-20O9S04579

3 NCTA-2009S05023

56

48

No information available No information available No information available No information available

No information available No information available No information available No information available

62-

62,16*-

6

6

100

100

24

36

applicant.

" Per information received, the score was changed due to "innovative element of the project, project unigueness". However, two out of five evaluators commented (in GMIS) the project as "not innovative", "internationally accepted method", only one pointed out "innovative application".

• The results of our sample testing show: project no.

NCTA-2008S00431

NCTA-2008S02850

NCTA-2009S03191

NCTA-2009/2S05853

NCTA-2009/2S06525

71

71

61

68

68

71,4

81,4

84,2

74,66

96,33

number of evaluators

No information was available

No information was available

No information was available

3

3

evaluator's highest score

No information was available

No information was available

No information was available

80

125

evaluator's lowest score

No information was available

No information was available

No information was available

64

64

preliminary scores minimum score to be accepted average score received

final scores

42

atlatszo.hu 2014.06.27.

26. oldal

Draft report - only for discussion

average score approved

77,2

81,4

84,2

74,66

96,33

5

5

5

3

3

evaluator's highest score

100

90

125

80

125

evaluator's lowest score

69

68

36

64

64

due to monitoring visit

N/A the scores have not changed

N/A the scores have not changed

N/A the scores have not changed

N/A the scores have not changed

number of evaluators

reason for change of scores (if applicable)

Demnet • "Cut-off" test (minimum score to be accepted - 7 8 ) :

project no.

NCTA-2009/2C06293

NCTA-2009/2C06313

NCTA-2009/2C06600

NCTA-2009/2C06710

NCTA-2009/2C05774

NCTA-2009/2C06537

preliminary scores average score received

82,6

77,6

73,8

78,6

80,6

79,6

number of evaluators

5

5

5

5

5

5

evaluator's highest score

125

100

125

125

125

100

evaluator's lowest score

48

48

36

8

18

18

average score approved

82,6

77,6

73,8

83,6

80,6

79,6

reason for change of scores (if applicable)

N/A

N/A

N/A Garas changed

N/A

N/A

final scores

scores: 60 to 85 the reason of changes is not documented

• Sample projects Sample sub-projects sub-project no. preliminary scores minimum score to be accepted

NCTA-2008C01958

NCTA-2008C00971

NCTA-2009C03941

NCTA-2009C04020

NCTA-2009/2C06413

89,5

89,5

68

68

78

average score received final scores

91

91,5

68,2

93,25

85

average score approved

88

86

68,2

93.25

85

number of evaluators evaluator's highest score

2

2

5

4

5

88

97

125

125

125

evaluator's lowest score 88 75 reason for change of scores Csanady Nizâk changed (if applicable) changed scores from scores from 9 4 to 8 8 8 6 to 7 5

43

8

48

36

N/A

N/A

N/A

atlatszo.hu 2014.06.27.

27. oldal

Draft report - only for discussion

Karpâtok •» " C u t - o f f " test (minimum score to be accepted - 4 8 ) :

project no.

NCTA-2009/2K06120

NCTA-2009/2K06542

NCTA-2009/2K05771

NCTA-2009/2K05990

NCTA-2009/2K06664

NCTA-2009/2K06081

47,8

43

42,8

51,2

50,4

48,6

5

5

5

5

5

5

100

125

100

125

80

80

4

1

1

8

16

8

47,8

43

42,8

51,2

50,4

48,6

preliminary scores average score received number of evaluators evaluator's hiqhest score evaluator's lowest score final scores average score approved reason for change of scores (if applicable)

N/A

N/A

N/A

N/A

N/A

N/A

• Sample p r o j e c t s Sample-sùb-projects sub-project no.

NCTA-2008K01669

NCTA-2008K02994

\\. '

NCTA-2009K04065

NCTA-2009/2K05523

NCTA-2009/2K06004

preliminary scores minimum score to be accepted

71

71

64,8

48

48

average score received

72,59

73,8

77,4

53

60,6

72,59

73,8

77,4

53

60,6

final scores average score approved number of evaluators evaluator's highest score evaluator's lowest score reason for change of scores (if applicable)

5

5

5

5

5

85

94

125

125

100

54

43

18

9

27

N/A

N/A

N/A

N/A

N/A

44