Walking the Talk - USAID Paper on Localization

WALKING THE TALK— REFLECTIONS ON THE USE OF INDICATORS TO MEASURE AND FOSTER LOCALLY LED DEVELOPMENT

Prepared in cooperation with PSC’s Council of International Development Companies

By Larry Cooley, Jean Gilson, and Indira Ahluwalia

Introduction

This paper builds on our prior paper, Perspectives on Localization,1 written under CIDC auspices and, to a lesser extent, on a second paper, Localization in Conflict Contexts, authored by MSI.2 Rather than repeat the arguments and lessons of those papers, it extends that discussion to specific considerations relevant to USAID’s proposed indicator for tracking its localization target #2: by 2030, 50 percent of USAID’s programming will place local communities in the lead to co-design a project, set priorities, drive implementation, and/or evaluate the impact of its programs.

The underlying goal of localization target #2 is to advance from inclusion of local actors to decision making by local actors and to measure that progress. This objective acknowledges that local communities, entities, and stakeholders have strategic insights, context, and know-how about what solutions are scalable and sustainable, and they have the legitimacy and standing to make lasting change. Target #2 also positions equity as the basis on which locally led development is applied that is, by integrating local actors with international practitioners in USAID programming to jointly make decisions about program design and development.

Localization Targets

Moving forward, USAID will provide "at least a quarter of all of our funds directly to local partners" over the next four years. And by the end of the decade, "50 percent of our programming, at least half of every dollar we spend, will need to place local communities in the lead" to set priorities, design projects, and evaluate the impact of USAID programs.

We endorse the letter and spirit of this second localization objective. Although the authors come from the development consulting community and draw principally on that perspective in framing our views, this paper, like the previous two, is intended as our contribution to an ongoing discussion, not as an advocacy document. It is informed by our experience helping USAID and its partners design and implement previous indicator programs and reform efforts. The paper’s Annex offers a number of comments and case illustrations contributed by CIDC members relating to USAID’s draft list of good practices.

Samantha Power, quotations from her speech at Georgetown University, November 4, 2021

We are submitting this paper having followed USAID’s public statements about the forthcoming indicator but prior to seeing the proposed indicator and accompanying guidance. Some of our comments may therefore be overtaken by events. We nevertheless decided to present them now in hopes that they might have implications for the testing and modification expected to take place in the coming months.

While CIDC provided useful input based on members’ field experience, this paper has not been formally vetted by CIDC and expresses only the views of the authors.

1 https://www.pscouncil.org/a/Resources/2021/Perspectives_On_Localization.aspx

2 https://www.msiworldwide.com/sites/default/files/2022-07/Localization_in_Conflict_Contexts_FINAL.pdf

Reflections on the Choice and Use of Indicators The Problem with Indicators Is That They Work

From our perspective, USAID’s experience suggests that indicators and performance measurement are very influential, but not always in the ways originally intended. This is partially because of the understandable tendency of respondents inside and outside the Agency to “teach to the test.” In this case, there is little doubt in our minds that the indicator selected will become the de facto, reductionist definition used by many to encompass the concepts of local voice, local ownership, and local leadership.

We have observed that performance measurement in USAID, as in other organizations, is typically designed to serve twin objectives learning and accountability. Sometimes these functions complement one another, but more frequently they don’t; and when there are trade-offs, accountability often crowds out learning. Given the nuance, contextual complexity, and variation in current debates about “locally led” development, we believe that fact calls for a focus, at least in the near term, on listening and learning as our community (donors, implementing partners, and the full range of local actors) continues to refine in operational terms the ways in which local leadership, experience, and voice are most effectively manifested and supported.

The Primacy of Upstream Action and Engaged Dialogue

As contractors, we typically support USAID programming in the implementation phase. The opportunities for local leadership in this phase, while significant, are smaller and fewer than those that are potentially available “upstream,” when country priorities and strategies are established, and interventions designed. This phase begins with the appropriation and budgeting processes, when decisions are taken that shape and limit the space available for local voice and prioritization; but considerable latitude also exists during program design.

We have also seen good use made of annual learning events and other participatory or country-led evaluative processes as mechanisms for local engagement and leadership, but only when those events are coupled with an appetite for incorporating the learning that emerges from such sessions back into the implementation process and future programming possibly in the form of rolling workplans.

In USAID’s draft list of “good practices,” five of 30 suggested actions relate directly to priority setting and design. Another four relate to evaluation At a minimum, we believe that such practices should be afforded special significance given their chances of affecting major resource allocation and programming decisions. Although earmarking, Congressional reporting, and Administration policies limit the scope for this kind of engagement to fundamentally shift resource allocation levels and priorities (in comparison to private foundations and grantsbased entities such as the Inter-American Foundation), we would argue that local leadership is greatly facilitated by delegating as much authority as possible to Missions to make resource allocation decisions. Absent genuine willingness and latitude by donors to reflect and adapt to local priorities and knowledge, our experience suggests that extensive local consultation processes can result in inflated and unrealized expectations, levels of effort disproportionate to their impact, and frustrated participants.

At the implementation stage of the project cycle, we have seen a variety of cases in which efforts to demonstrate local leadership in the absence of budgetary and programming latitude yield empty and overlapping consultations. In the contracting sector, for example, this includes cases where competitive RFPs result in frantic efforts by a host of bidders to engage the same array of host country actors during a compressed proposal development timeframe. Ironically, the recent preference in some USAID procurements for engaging local partners on a non-exclusive basis has, on some occasions, compounded this problem.

Selecting an Indicator

There are important differences between localization targets #1 and #2. Most importantly, we believe there are major risks in the application of a metric for target #2 that lacks flexibility to reflect important contextual factors such as fragile versus non-fragile states, authoritarian versus democratic governments, and inclusive versus exclusive treatment of marginalized groups. It is also unrealistic to expect a static indicator to keep pace with and reflect the reality of an evolving local landscape, and drive progress in places where local leadership and transformation are proceeding at breakneck pace

Definitional issues surround even relatively simple concepts such as “local implementing partner.” But these challenges pale in comparison to the nuances associated with concepts such as “local communities,” “in the lead,” and “shifting power.” Underestimating this complexity could spur a cottage industry of definitional debate, inadvertently further privileging the voices of entrenched elites, given where power resides, and inadvertently incentivizing actions that would not enjoy widespread support as strategies for enhancing local leadership.

One approach to measurement would be to use “good practices” applied by USAID during the project cycle as a proxy for assessing progress toward locally led development. Based on experience with other indicators that use the enumeration of internal administrative practices as a proxy for intended outcomes, we believe this approach has some merit in focusing USAID’s attention on concrete, manageable actions it can take unilaterally. But it could also be seen as reflecting a check-the-box mentality, failing to reflect local perspectives regarding these measures, and encouraging a reluctance to consider alternative ways forward.

Language also matters. Problems cited to us include a range of embedded and often inadvertent messages in how we even address equity, inclusion, agency, and the distribution of power (for example, “we place communities in the lead”).

A final and related concern is that any indicator and assessment system established and applied uniformly by a single donor runs an inherent risk of inadvertently fragmenting measurement at the country level and thereby undermining the concept of locally led development it is intended to advance

We recognize that there are trade-offs between the considerations we raise above and the action-forcing power of a single, unambiguous indicator—particularly one for which USAID hopes to achieve global support. We also appreciate the practical obstacles to introducing an indicator that imposes additional burdens on USAID’s already-stretched staff in Washington and in partner countries.

The remainder of this paper offers our suggestions on how to maximize these upside benefits and minimize the downside risks. It incorporates lessons from, among other things, our experience working alongside USAID in the roll-out of the Development Fund for Africa, environmental impact requirements, and policies on CLA, and recent efforts we have observed from GiZ and OECD/DAC.

Getting the Most Out of an Indicator and the Indicator Process

If USAID decides to adopt a single, uniform indicator that can be centrally managed and applied in a consistent manner, we believe the best options to maximize benefits and minimize risks relate to the process and the scaffolding that surround whatever indicator is selected.

Some ideas we have seen used to positive effect include:

Process

• Country-Based Coordination: If USAID is to achieve widespread acceptance of the indicator, it should take full advantage of existing consultative mechanisms established by host governments and others to limit cumbersome, donor-specific, and duplicative consultation. Where possible, these same venues can be used to establish agreed standards and examples of locally led development.

• Self-Assessment with Selective Audit: USAID’s experience underscores the fact that the more Missions and their key stakeholders are involved in designing and assessing the consultative and power-shifting processes, the greater their impact will likely be. Alternatively, the more these processes are seen as centrally dictated bureaucratic requirements, the more likely they are to lack local ownership and to be ignored, or even resented, at the Mission. In our view, the most realistic way to square that circle is to ask Missions to score themselves against established criteria, to present the rationale for that self-assessment, and to be subject to selective audit. This process has been used successfully by USAID on several previous occasions, often linked to Mission-based task forces and reform initiatives. In several of those cases, Regional Bureaus played important intermediation roles.

• Local Advisory/Review Groups: Missions should be encouraged and supported to establish stakeholder consultation processes where these do not exist and to use these processes to advise on programming and to push the envelope of opportunities for local leadership. Among other things, these groups could be asked to express their opinions on USAID’s self-assessment ratings, or perhaps provide their own ratings.

• Peer Learning: Missions could be encouraged to self-nominate to pilot innovative approaches for the measurement of locally led development, with high-level recognition and an annual process for global information sharing such as the Spring Reviews USAID used to run so successfully.

• Country-Based Learning Events: USAID should actively encourage, participate in, and (if necessary) help fund country-based learning events that feature local voices and include explicit discussion of issues of localization, inclusion, and decision making, with feedback loops or other mechanisms to ensure learning is incorporated into country strategies.

Scaffolding

• Maturity Models: While admittedly more complicated than single, binary indicators, USAID has had positive experience with the use of maturity models to track and support the evolution of multifaceted and context-

dependent processes such as localization. The use of these models could be either a substitute for, or an addition to, the use of a simple indicator.

• Supporting Narrative: Again, at the cost of some additional effort, Missions could be asked to provide supporting narratives describing what the Mission or its individual activities have done to support locally led development. These narratives, along with the results of simpler indicators, could form the basis for discussion inside the Mission, with local stakeholders, and with Washington.

• Case Competitions: USAID has done some of its most innovative work and advanced some of its most lasting internal reform though peer-to-peer learning. In several recent and not-so-recent examples, the Agency uses case competitions to accelerate, enhance, and systematize this learning process. The roll-out of localization target #2 would be well suited to a process of this sort, perhaps associated with high-level recognition and more tangible benefits for the best entries.

• Indicator Reference Sheets: Considerable supplementary guidance to Missions could be embodied in moreextensive-than-usual indicator reference sheets that, in addition to the typical definitional guidance, encourage complementary actions of the sort suggested in this paper and elsewhere.

Supporting the Internal Change Process

As USAID leadership has rightly emphasized, implementing the current localization initiative has profound implications for the Agency’s programming and operations, talent acquisition and development, organizational culture, and the day-to-day work of USAID staff. Having watched and participated in previous reform efforts, we have deep respect for the demands associated with this effort, and we hope USAID’s implementing partners will find effective ways to support it.

Most of the draft list of good practices refer to actions to be taken directly by USAID staff and we wholeheartedly support USAID’s efforts to get Missions the staffing and other resources they need to implement the letter and spirit of the localization effort.

But there are several areas where the Agency’s partners can do more to help. In addition to the work undertaken by contractors and grantees during activity implementation, this potential support includes upstream assistance as a force magnifier in organizing consultative processes (good practices 1-5); the design and implementation of participatory and community-led monitoring systems (good practices 23-24); strengthening the readiness of local organizations to take on key leadership roles (good practices 26 and 29); and designing and implementing participatory evaluations and learning events (good practices 27, 28, and 30).

If it would be useful to USAID, we are confident that CIDC members could contribute additional examples of these practices in action. Current implementing partners can also serve as eyes and ears for USAID staff, providing observations and insights into some of the challenges being faced during roll-out of the reform.

We applaud USAID’s intensified focus on locally led development and the Agency’s recognition of the power of indicators to drive outcomes. It is our hope and belief that this impact can be substantially strengthened by incorporating additional contextual considerations and stakeholder engagement in shaping how these measures are operationalized in different settings.

Annex: Good Practices—Select Comments and Examples3

1. Listening Tour

We have found that careful selection of “community-level local actors” is critical to avoid bias and to access diverse perspectives. Finding/constructing a safe, effective, and locally credible opportunity for the most vulnerable of these actors to raise their perspectives, concerns, and priorities is particularly challenging.

Our team in Kenya was able to conduct a well-focused listening tour aimed at key target groups. This was possible because the program focused on a single county with a target of Masai people typically under-served by the health system. Insights gained from talking to Masai community members were then reflected in the design of activities as well as the requirements that the project included in an RFA. In Mali, there was no opportunity for a true listening tour as part of our co-design activities. When we subsequently conducted KIIs and small group discussions, what emerged was largely a list of things local stakeholders wanted from the project that were outside of the scope of what the project was contractually able to provide. Contrasting these two experiences, our lessons learned are: conduct listening tours as early as possible and with a clear target group; if engaging later in the process, pose focused questions relevant for the established scope of the project; and be mindful of raising expectations to avoid creating frustration.

3. Co-design with Local Communities

We have seen significant confusion in cases where clear guidance was not provided regarding the implications of co-design for eligibility to compete for and receive resulting awards. Used properly, however, we have found that co-design is an opportunity to “de-center” USAID/donor/IP perspectives, especially when those codesign efforts are designed, led, and implemented by local stakeholders and FSNs.

The most meaningful consultations we have been involved in begin with co-defining the problem to be solved, not just stating the problem and co-designing solutions. In those cases, USAID needed to be prepared to cede the prioritization of problems to local decision makers.

Our experience is that co-design sometimes gives the false impression to communities that they have more power than they actually have over how programs are designed. Where co-design is used, relevant earmarks and other parameters should be carefully explained, any needed USAID approvals should be made explicit, and relevant restrictions (regarding construction and procuring certain commodities, for example) should be explained.

We have found that projects that allow for more local input, voice, and ownership don’t always align to USAID’s standard indicators. For example, on a health program we helped implement, the Mission disagreed with the local definition of the problem and some of the proposed solutions and suggested that the project encourage co-creation of solutions more closely aligned to standard indicators. With encouragement from USAID/Washington, the decision was ultimately made to proceed with the locally led solutions and those solutions remained in use after the project ended.

3 These comments are drawn from contributions submitted by CIDC member companies in response to a three-day-turnaround data call by CIDC. The authors have anonymized the submissions (“we”) and edited for length and consistency. Comments address some, but not all, of the “good practices” noted in USAID’s draft list.

4. Consultation and Joint Finalization with Government Partners of Activity Results Through the Design Phase

As implementing partners, we often begin a new project only to find that key government counterparts have never heard of or have misunderstood the nature of the project. To establish a common understanding, it helps enormously when USAID organizes an effective launch process with government officials. As a minimum, USAID should provide information to IPs about who in government was engaged in the project design and facilitate contact between IPs and those individuals as soon as possible after implementation begins.

We know of several cases in which USAID prioritized local measurement and ownership of results by using and reinforcing national information systems to measure change rather than project-based information systems that end with the end of a project. This approach provided less control to USAID but contributed to greater local ownership and sustainability.

We have found that incorporating qualitative, in addition to quantitative, results measures helps to incorporate more local voice and nuance.

5. Local Leadership Councils/Advisory Boards

We have found the Tips document on political economy analysis (PEA) and the concepts of thinking and working politically (TWP) to be very useful in guiding our identification of, and engagement with, local leaders and leadership structures.

CIDC members cited various cases where advisory boards provided a strong local voice and valuable input. They noted several instances in which such boards aided communications, facilitated approvals, and bolstered relationships with the government and other stakeholders critical for implementing and sustaining project activities. They observed, however, that these mechanisms sometimes introduce confusion about accountability, and that these boards are often most effective when USAID cedes some control regarding the identification of board members and participants.

Members also reported working successfully with a variety of other local consultative structures such as local clinic management councils and private sector advisory boards to improve program targeting , governance, quality control, and social accountability.

6. Requests for Information (RFIs) Targeted to Local Entities

Several CIDC members noted cases where it was cost-prohibitive for many local partners to participate in RFIs or consultation processes without the assurance of funding.

CIDC members cited and praised cases where local actors had numerous opportunities to hear about an RFI through multi-message, multi-channel delivery mechanisms and sessions hosted by USAID or IPs for local organizations on how to respond to RFIs.

Expressions of Interest were mentioned by several CIDC members as a relatively simple way for local organizations to share their perspectives and signal their potential interest in engaging in specific activities.

7. Unsolicited Proposals or Applications from Local Entities

We have seen numerous cases in which unsolicited proposals generated by local organizations and presented directly to USAID resulted in little return for USAID or the organizations submitting. If this mechanism is

promoted, our experience suggests that it will be important to provide clear guidance to those interested in submitting, a transparent account of how unsolicited proposals are handled, and statistics related to awards.

8. Co-creation of an Award with Local Entities

Co-creation is often more time consuming than anticipated. It is also difficult to execute in a competitive environment where competing local actors may be reluctant to share detailed information.

We frequently use APS mechanisms within active projects to solicit ideas and establish a co-creation/design exercise for local subcontracts/subgrants. This is often an iterative process to help local organizations think through their business/development strategies and position themselves to receive funding.

9. Limiting Competition or Eligibility to Local Entities

We have found that many local partners who understand the requirements associated with being USAID prime recipients prefer to engage as a sub so they can focus on technical work. CIDC members cite several examples where major local organizations initially wished to move from subgrantee or subcontractor to become prime recipients but had difficulty with the additional requirements and ultimately decided not to do so.

Our experience suggests that USAID needs to be very explicit about how it is defines “local” in any given solicitation. Even using its ADS definition of “local entity” leaves room for interpretation on issues such as whether an organization is truly independent from a parent or confederation. USAID may wish to consider creating a mechanism whereby prospective bidders can confirm prior to preparing a proposal whether or not they will be considered “local” for purposes of that specific solicitation.

We have been involved in several situations where IPs were effectively engaged to provide technical assistance to local awardees that faced initial implementation challenges on issues related to compliance and MEL.

10. Solicitations in Local Language(s)

We have seen significant benefits in those few cases where applicants were able to submit their applications in local languages. This implies, however, that the merit review committee needs to be fluent in the local language(s) and that the resulting award needs to be translated into local language

11. Industry Days Targeted to Local Entities

Several CIDC members reported positive experience with targeted events for potential partners as well as with “mixed” industry day events that provide opportunities for learning and networking.

14. Transition Award

We have witnessed several situations where, in hindsight, there was inadequate discussion with potential transition award recipients regarding the requirements for being a prime awardee and the support available.

One contractor cited a case where, over a four-year period, it provided direct support to a local organization to strengthen its capacity for direct award and enrich the overall program. In Year 1, a GUC helped to launch a key activity, with 5 percent cost-share; in Year 2, a GUC supported follow-on activity with 15 percent costshare; in Year 3, a GUC included 55 percent cost-share by the local partner. The project also helped the

partner to develop an activity concept for which the local organization secured non-USAID funding, which it administered directly.

Another contractor cited two projects that included contract-mandated requirements to prepare local partners for direct USAID transition awards. Despite that preparation, in both cases funding was not ultimately available, and the partners did not receive a direct award.

A third contractor referenced a project that included a requirement for local partners to graduate and become USAID primes. Many key actors in the local market system did not want to become USAID primes and some of those that could potentially become primes were not the well suited for that role.

17. Government-to-Government (G2G) Agreement

CIDC members noted challenges faced in G2G agreements but also cited cases they had been involved in where G2G arrangements had been used effectively and where they had provided technical assistance in assessing government systems for readiness to receive funding, co-creating milestones and monitoring plans, and helping to implement a G2G agreement alongside government counterparts.

19. Award to Local Organization with Overhead Costs

Several CIDC members voiced their support for factoring in realistic overhead costs for local organizations, without which these organizations are unable to grow and to put in place the systems needed to comply with USAID regulations. The 10 percent de minimis allowance is often insufficient, they noted, and they cited cases of support provided to ensure that reasonable allowances for the indirect costs of local partners were included in USAID and non-USAID projects. This support sometimes includes help to local organizations in calculating indirect costs rates and establishing systems to budget and account for the use of these funds.

20. Mid-implementation Co-Design with Local Partners/Sub Awardees

We have observed that global/regional buy-in platforms are much more likely to incorporate co-design/cocreation processes than are conventional bilateral projects.

We have not generally used co-design with local partners due to the relatively short period of performance for most of our activities.

21. Use of Locally Led Monitoring Indicators

We use the CBLD-9 indicator internally to track our performance in augmenting local partner capacity, but we have found that CBLD-10 is sometimes hard to use because commercial entities do not wish to share proprietary financial information. We recommend that USAID be more explicit in its request for CBLD-9 and CBLD-10 indicators in its solicitations and work closely with local partners to encourage more complete and consistent collection and reporting of this data.

24. Pause and Reflect with Local Stakeholders Using the Locally Led Development Spectrum

When we employed a community-led, CLA approach with periodic pause and reflect sessions, community members reported a heightened sense of agency, and project staff appeared to be more committed to generating and sharing evidence on what they had learned.

25. Community-Led Monitoring of Programming

We have made the mistake of trying to incorporate indicators into community-led monitoring systems that turned out to be extremely complicated for us to measure, let alone for local partners to measure. This problem could have been avoided if we had developed context-relevant indicators in collaboration with local stakeholders.

26. Capacity Strengthening Efforts Aligned with the Local Capacity Strengthening Policy

We have seen relatively few cases of deep collaboration between USAID and existing IPs focused on implementation of the Local Capacity Strengthening Policy. We believe that much could be gained by such collaboration, including identifying trainings that groups of local partners and potential partners could attend with support of IPs who know USAID well.

We utilize the Organizational Performance Index (OPI) and Organizational Capacity Assessment (OCA) to measure changes in performance of grantees over time, assess grantee capacity, and co-design Capacity Improvement Plans.

We facilitated the New and Underutilized Partner Pre-Award Survey (NUPAS) process to diagnose organizational strengths and weaknesses, identify areas where award conditions would be required, and then to strengthen the capacity of each organization to address areas that would have prevented them from receiving direct USAID awards. Within two years, organizations moved from being high-risk to no-risk as evidenced by the NUPAS assessment. Organizations successfully reconstituted and expanded their boards, improved their internal control systems, and purchased accounting software that helped them generate financial accounts and reporting that were compliant with USAID regulations and reporting requirements.

We try to avoid providing project-based technical assistance. Instead, we provide funding that allows local organizations to issue solicitations to procure expertise in their local market. We also promote peer exchanges and other technical assistance from organizations in the national or regional market and have facilitated a network of social marketing organizations to share best practices, develop joint proposals, and explore joint procurement.

28. Evaluation Leveraging Local Expertise

Our MEL team is currently creating guidelines on best practices for working with, and providing training to, a wide range of local evaluation partners.