70 percent of the funds given to the two largest U.S. contractors for a cash for work project in Haiti was spent on equipment and materials. As a result, just 8,000 Haitians a day were being hired by June, instead of the planned 25,000 a day, according to the IG.
Furthermore, HRI-II was not on track to complete all activities before the scheduled end date of September 2012. Budget line items in the contract provided for grants under contract ($37.3 million) and “non-grant under contract” activities, such as direct procurement of goods and services ($3.3 million) for a total of $40.6 million. As of February 2012, 141 activities worth about $22.9 million had been developed and approved, leaving $17.7 million available for new activities to be approved, implemented, completed, and closed in the 7 months left.
The audit also found some additional areas in which program management could be improved
as discussed in the following findings.
The monitoring and evaluation system was weak (page 3).
Community involvement in activities was not sufficient (page 7).
A short-term technical assistance activity did not meet expectations (page 7).
An environmental review for an activity was not performed and approved before
implementation (page 7).
This report recommends that OTI:
1. Implement a plan to measure and report progress toward meeting HRI-II’s goals for the remainder of the program (page 5).
2. Update its implementation plan for the remainder of the program to include estimated timelines for each phase of an activity for development, implementation, and closeout
(page 5).
3. Work with USAID/Haiti to conduct and document data quality assessments for the indicators OTI reports on for the mission’s operational plan and performance report (page 5).
4. Incorporate plans to engage communities when developing activities so the communities will sustain the efforts (page 6).
5. Implement communications plans into ongoing and new activities to make the local population aware of the activity and its benefits for the community (page 6).
6. Implement processes to assess short-term technical assistance with grantees throughout the grant cycle (page 7).
7. Review the existing environmental documentation of current and planned activities for compliance with environmental review requirements (page 9).
8. Require Chemonics to confirm that environmental reviews are completed before any activities begin (page 9).
Detailed findings appear in the following section. The audit scope and methodology are
described in Appendix I. Management comments are included in their entirety in Appendix II, and our evaluation of management comments is included on page 10 of the report.
AUDIT FINDINGS
Monitoring and Evaluation System Was Weak
USAID’s Automated Directives System (ADS) 200.6 defines performance management as the systematic process of monitoring the achievements of program operations; collecting and analyzing performance information to track progress toward goals; and using that information to influence how decisions are made and how resources are allocated. According to ADS 203.3.3.1, the performance management process has four principal steps: (1) establishing a performance management framework, (2) collecting and analyzing performance information, (3) using that information to influence decision-making and resource allocation throughout a program, and (4) communicating results. Missions use performance management plans as tools to plan and manage the process of monitoring, evaluating, and reporting progress toward achieving objectives. ADS does not provide a format for performance management plans; USAID offices and missions are allowed to use the format that best fits their needs.
OTI’s contract with Chemonics states, “Timely evaluation of activities and the program more broadly is a critical component of ongoing, updated analysis, which is accomplished through periodic strategic review sessions for regular evaluation of recent activities and the program.” Because OTI communicates frequently with its partners, it can adjust activities as needed throughout the course of a program. However, in the case of HRI-II, the lack of management tools to measure progress toward program goals made it difficult to measure the program’s impact. Specific monitoring and evaluation concerns are discussed below.
Program Management Was Not Comprehensive. HRI-II’s monitoring and evaluation focused primarily on the activity level and to a lesser extent on the program as a whole. Each activity had three levels of objectives and indicators to measure outputs, outcomes, and impact. HRI-II developed two guides, Activity Cycle and Guidebook and Activity Management Guide, to help employees manage the program. However, while the guides provided templates and addressed procedures for awarding and administering grants for the program, they did not contribute to monitoring the entire program.
According to the HRI-II contract, program monitoring efforts should include regular evaluations of the program, as well as the activities. These evaluations should allow the HRI-II staff to refine the program’s focus. The contract further states that the participants, nature, structure, and frequency of the evaluations depend on the country context, program needs, and the staff. In general, they should be conducted between two and four times a year.
OTI has conducted only one evaluation for HRI-II so far, and it took place in June 2011 to discuss and draft OTI’s strategy for Haiti, to build the new HRI-II team, and to propose ideas of where to conduct activities. OTI officials said they conduct quarterly “rolling assessments” to assess programmatic issues, program development, and implementation. In addition, OTI and Chemonics met weekly to cover program development and implementation, and to discuss problems, delays, and challenges.
While these meetings contribute to program development and implementation, they do not measure progress toward ultimate program goals. HRI-II Did Not Have Program-Wide Monitoring and Evaluation Plan. When the program started in March 2011, the staff developed a monitoring and evaluation guide that consisted of guidelines and procedures for developing monitoring and evaluation plans. However, it did not constitute a formal plan that HRI-II’s staff could use to track progress, nor did it explain how results would be measured and documented throughout the program.
Chemonics officials said they decided not to prepare a formal plan because the activities were so different from each other that monitoring and evaluation plans needed to be tailored to the activity level. At the time of the audit, HRI-II had 141 activities in progress. Because each activity had its own individual monitoring and evaluation plan, Chemonics was tracking 141 different plans, but had no single plan for the entire program. After an activity was completed, the Chemonics staff performed a final evaluation. OTI and Chemonics officials said they would use final evaluations, comparisons of “clusters” of similar activities, and “thematic reviews to provide lessons learned and a feedback loop” to guide future activities. However, given HRI-II’s short time frame and the different nature of the activities, this approach might not provide useful or timely information as a monitoring and evaluation tool.
Poorly Defined Indicators. USAID guidance states that performance indicators should be unambiguous about what is being measured and closely track the results they are intended to measure.
Some of the performance indicators Chemonics developed were not well-defined. One activity provided computer equipment, chalkboards, benches, chairs, desks, and school kits consisting of backpacks and school supplies to two public schools in target communities. Yet a performance indicator for that activity measured the number of students who returned to school. OTI developed the activity to help offset some of the costs borne by the parents in sending their children to school, but the performance measure did not correlate to the activity.
Other activities reviewed had only one or two indicators, and they were poorly defined; for example, one purportedly measured Number of government institutions supporting a conflict and/or fragility. An activity that provided a local municipality with an engineering study for improving one of the town’s main roads was measured by Number of reconstructed national governing institutions and systems that receive USG [U.S. Government] assistance to incorporate principles that support democracy and government legitimacy. None of these activities had indicators that clearly measured objectives.
Implementation Plans Lacked Timelines. OTI requires Chemonics to track activity data through OTI’s activity database, which includes a field for a detailed implementation plan. In addition, an activity or grant notes field provides chronological updates of the activity’s progress or anything discussed in implementation meetings. The combined information in the two fields helps OTI track an activity’s progress.
However, the detailed implementation plans for some of the activities reviewed did not have enough information for OTI to see whether the activity was on track to end on time and to meet its objectives. For example, the plan for providing Haiti’s Parliament with temporary offices and meeting space consisted of “The subcontractor will be responsible for the following:
1) Assembling and installation of steel-framed structures; 2) Connection of utilities.” No dates or estimated timelines were included.
Data Quality Assessments Not Performed. ADS 203.3.7.3 states that performance data reported to USAID/Washington in the performance plan and report must have had a data quality assessment (DQA) completed within the past 3 years. The goal of a DQA is to inform managers of data strengths and weaknesses and the extent to which data can be relied upon to influence management decisions.
Although results of OTI’s HRI-II activities were reported in USAID/Haiti’s fiscal year 2011 performance plan and report, the staff did not conduct DQAs of the data for the five indicators that it reported on in 2011.
All of the problems discussed in this finding stemmed from the fact that OTI did not make monitoring and evaluation a priority early in the HRI-II’s implementation. OTI officials said it started becoming a priority when Chemonics hired a full-time monitoring and evaluation officer in August 2011.
According to an OTI management review completed in January 2011, the HRI I program did not place enough emphasis on monitoring and evaluation. The team conducting the review recommended that OTI hire a permanent staff member to oversee monitoring and evaluation. OTI advertised for a foreign service national monitoring and evaluation specialist in April 2011, but officials said they did not receive any qualified applicants.
Under the program’s monitoring and evaluation framework, it was hard to consistently and effectively evaluate how results and targets were measured. This handicap could result in activity objectives not being met. To strengthen the program’s monitoring and evaluation system, we make the following recommendations
Recommendation 1. We recommend that USAID/Office of Transition Initiatives implement a plan to measure and report progress made toward achieving program goals for the remainder of the program.
Recommendation 2. We recommend that USAID/Office of Transition Initiatives require Chemonics International to update its implementation plans for the remainder of the program to include estimated timelines for each phase of an activity for development, implementation, and closeout.
Recommendation 3. We recommend that USAID/Office of Transition Initiatives
coordinate with USAID/Haiti to conduct and document results of data quality assessments for the indicators that USAID’s Office of Transition Initiatives reports on for the mission’s operational plan and performance report.
Communities Were Not Involved
Enough in Activities
As described in USAID’s sustainable development strategy, “Sustainable development mandates participation. It must involve, respond to, and be accountable to the people who will live with the results of the development effort.” According to the HRI-II contract, the goal of USAID/OTI’s efforts in Haiti is to help stabilize the country while laying the foundation for development and good governance. The contract further states that OTI would pursue all programming with an eye to community participation, government engagement, and alignment with the unified U.S. Government approach in Haiti.
HRI-II’s offices in Port-au-Prince, Cap-Haitien, and Saint-Marc have worked with local communities to develop activities tailored to meet their needs. Nonetheless, not all of the activities implemented in the corridor around Cap-Haitien were developed by the regional office that understood the social and economic context of the area, and not all activities implemented have involved community participation in a way that guarantees sustainability. OTI’s 2012 Management Review pointed out that the leadership of the Chemonics office in Port-auPrince often failed to communicate to the Cap-Haitien team the information necessary to conduct activities there that were developed or managed by the Port-au-Prince office. Program officials expressed concerns regarding activities planned by the Port-au-Prince office that did not utilize local knowledge. In addition, they said Chemonics used contractors from Port-auPrince to implement a number of activities in Cap-Haitien and Saint-Marc; these contractors brought their own people to do the jobs instead of hiring locals. As a result, residents saw jobs in their neighborhoods being done by outsiders, and without an understanding of the activities, they did not see how anyone local benefitted.
Program officials said that for some activities, particularly in the Cap-Haitien corridor, community members did not feel engaged and did not take ownership of the completed projects. Urban beautification projects in the towns of Limonade and Caracol failed for these reasons. The purpose of these projects was to improve public areas by installing plants and benches, as well as doing minor masonry work, and to project “a positive image of what role the nearby Caracol industrial park and other upcoming economic investments will play in citizens’ lives.” The landscaping work in Limonade was not completed as planned because of a lack of coordination between two HRI-II activities. Chemonics purchased and planted some seedlings for the town center, but they died from lack of care. According to the project’s final evaluation report, residents did not understand how the activity led to the beautification of the area nor did they associate it with the industrial park. Limonade’s mayor said the municipality could have been involved more in planning the activity to ensure its success.
Although many of OTI’s activities included community collaboration in development and implementation, some did not, and they were not very successful because OTI did not always incorporate plans into activity designs to promote community ownership and sustainability. Furthermore, OTI did not require Chemonics to implement a communication plan for each activity to increase the local population’s awareness of it and its benefits to the community. While OTI must strike a balance between coordinating closely with local communities and meeting its mandate of being fast and flexible, activities implemented with limited participation from local communities can hurt the prospects for sustainability. Therefore, we make the following recommendations.
Recommendation 4. We recommend that USAID/Office of Transition Initiatives incorporate plans to engage communities when developing activities so that they will take ownership and sustain the efforts.
Recommendation 5. We recommend that USAID/Office of Transition Initiatives require
Chemonics International to implement communications plans for ongoing and new activities to increase the local population’s awareness of the activities and how they will benefit the community.
Short-term Technical Assistance
Activity Did Not Meet Expectations
The audit team reviewed one of nine ongoing short-term technical assistance activities. These
provide an organization with an expert who trains or mentors employees for a short period of time. The HRI-II activity provided Haiti’s national mapping agency, Centre National de l’Information Geo Spatiale (CNIGS) with a geographic information system (GIS) specialist for 6 months because the agency lost most of its management staff, facilities, and equipment during the earthquake. The consultant was responsible for helping transfer mapping information from international humanitarian agencies to CNIGS and for working with other donor agencies in developing an information system for coordinating housing and neighborhood reconstruction. According to CNIGS and the consultant, the information system was not finished when her contract ended in February 2012. CNIGS’s acting director said she did not provide the help the agency needed and her input did not meet its needs. The acting director added that the consultant’s efforts to document user requirements for the information system were insufficient and would have to be redone by the system developer. The consultant said she had difficulties getting CNIGS staff members to discuss data sharing agreements, which delayed establishing the data-sharing protocols. In addition, the tasks of implementing the conceptual model of the information system and configuring the GIS server were only halfway completed by the end of her contract.
The activity did not meet its objectives because Chemonics did not communicate effectively with CNIGs or provide adequate support to the consultant. In October 2011, shortly after the consultant began working at the agency, she told Chemonics officials she was having difficulties discussing with CNIGS what they expected her to do in developing the system. Chemonics officials told her to continue to try to work with the agency.
Chemonics officials said they had separate meetings with the consultant and CNIGS monthly, but they focused on program development. The Chemonics officials said they never asked agency staff members whether the consultant was meeting their needs and learned only at the activity’s end that CNIGS was not satisfied with her. Short-term technical assistance might not help grantees if they do not get the type of assistance they need or if they do not use the expertise provided appropriately. To assist OTI in managing current and future short-term technical assistance grants, we make the following recommendation.
Recommendation 6. We recommend that USAID/Office of Transition Initiatives implement processes to assess short-term technical assistance with grantees periodically throughout the grant cycle.
Environmental Review Not Performed and Approved Before Implementation
ADS Chapter 204, “Environmental Procedures,” states that USAID must fully comply with environmental procedures established in the Code of Federal Regulations (22 CFR 216). USAID is required to “integrate environmental issues into its programs, both to meet U.S. Government legal environmental obligations and to optimize economic and social development results.”
Supplementary guidance to ADS 204 provides descriptions of environmental procedures to include in requests for proposals and award documents. The purpose is to give USAID time to conduct environmental reviews and incorporate environmental factors and mitigating measures in the design and approval of each program and activity before making an irreversible commitment of resources. ADS further states that the review should be done as early as possible in the design process to allow enough time for more detailed subsequent environmental review and concurrence, and for integrating environmental mitigations into the process.
Chemonics has established procedures for environmental reviews to screen each activity for potential adverse impacts, recommend determinations to categorize environmental risk, and identify mitigation and monitoring measures. The procedures for mitigation and monitoring include templates for environmental mitigation plans and reports; these were developed for USAID/Haiti to use when implementing infrastructure and related activities. Several of the activities the team reviewed were designated as “negative with conditions,” which applies to activities that could have moderate adverse environmental impacts, but could be managed effectively with appropriate mitigation and monitoring measures. For these activities, Chemonics used plans that included mitigation measures and reporting on monitoring of those measures.
In one of the activities, OTI did not complete the required environmental review and mitigation and monitoring plan before implementing the activity, which was designed in June 2011 to generate temporary employment opportunities through planting 700,000 jatropha seedlings in the Saint-Marc corridor. In Chemonics’s environmental review of the activity, it recommended a determination of “negative with conditions,” requiring an environmental mitigation and monitoring plan.
However, although the seedlings were planted in August 2011, OTI had not approved the environmental determination, and the mitigation and monitoring plan was not in effect. The environmental officer visited the jatropha plantation in August 2011 and said that no negative effects had occurred; the officer said Chemonics had done scientific research to be sure that the type of jatropha was proper for the area, but had not yet documented the environmental work performed. Subsequently, Chemonics developed a mitigation and monitoring plan and submitted it to USAID/Haiti and OTI for approval. The mission environmental officer and contracting officer’s representative approved the environmental mitigation plan and report in October and November 2011, respectively—after the jatropha had already been planted.
The lack of planning for environmental mitigations before implementation happened because OTI and Chemonics neglected to be sure that they complied with the requirements to complete the environmental documentation and to obtain the necessary USAID approvals before planting the trees. Potentially adverse environmental impacts can occur if proper mitigation and monitoring procedures are not put into place before implementing an activity and monitoring it. Therefore, we make the following recommendations.
Recommendation 7. We recommend that USAID/Office of Transition Initiatives review the existing environmental documentation of current and planned activities for compliance with environmental review requirements, and document the results of the review.
Recommendation 8. We recommend that USAID/Office of Transition Initiatives require Chemonics to complete environmental reviews before implementing activities.
EVALUATION OF MANAGEMENT
COMMENTS
Based on our evaluation of management comments on our draft report, we have determined that management decisions have been reached on all recommendations. In addition, final action has been taken on Recommendations 2, 4, 5, 6, 7, and 8. Our detailed evaluation of management comments follows.
Recommendation 1. OTI agreed to implement a plan to measure and report progress toward meeting program goals for the remainder of HRI-II. Officials said they would hire a local contractor to conduct third-party monitoring and evaluation. In addition, OTI plans to require the monitoring and evaluation contractor to develop indicators to be used for measuring and reporting on goals for the remainder of the program. OTI expects to have this in place by December 31, 2012. Based on OTI’s planned actions and estimated time frames, a management decision has been reached on this recommendation.
Recommendation 2. OTI agreed to update its implementation plan for the remainder of the program to include estimated timelines for each phase of an activity for development, implementation, and closeout. In May 2012 OTI updated its procedures and required Chemonics to submit detailed implementation plans that include timelines for estimated completion of each phase of an activity. Based on OTI’s response and supporting documents provided, final action has been taken on this recommendation.
Recommendation 3. OTI agreed to conduct and document DQAs of the indicators OTI reports on for the mission’s operational plan and performance report. In management comments, OTI officials said they would work with USAID/Haiti’s Program Office on the DQAs, and they expect to complete them by December 1, 2012. Based on OTI’s response and estimated time frames to complete the DQAs, a management decision has been reached on this recommendation.
Recommendations 4 and 5. To address the recommendations, OTI issued updated guidance to Chemonics in August 2012 that required it to incorporate community engagement and communication into each activity’s implementation plan. The plans should include information about how and when Chemonics plans to engage the communities in an activity and what communication tools it plans to use to increase activity impact by highlighting participation of local authorities and explaining the benefits to the community. OTI provided a copy of the updated guidance explicitly stating that community engagement is to be conducted in the activity development stage to promote community ownership and sustainability for all community stabilization activities. As a result of OTI’s response and completed actions, final action has been taken on these recommendations.
Recommendation 6. OTI agreed to implement processes to assess short-term technical assistance with grantees throughout the grant cycle. In response, OTI has implemented a procedure to include meetings with grantees as a monitoring and evaluation requirement for its short-term technical assistance activities. OTI provided copies of the updated monitoring and evaluation guidance issued to Chemonics that includes the new requirement. Based on OTI’s response and documented actions, final action has been taken on this recommendation.
Recommendations 7 and 8. OTI agreed to review the existing environmental documentation of current and planned activities for compliance with environmental review requirements, and to require Chemonics to confirm that environmental reviews are completed before any activities begin. OTI conducted a review of current and planned activities for environmental compliance. It also worked with USAID environmental officials to develop umbrella mitigation and monitoring plans for infrastructure, water and sanitation, and watershed activities. In addition, OTI added a step in its approval process to document environmental compliance requirements that must be completed before an activity can be approved. OTI provided documentation of the new requirement and a checklist documenting its environmental compliance review of the ongoing and planned activities. OTI also provided copies of the umbrella environmental mitigation and monitoring plans. Based on the actions taken by OTI and supporting documents, final action has been taken on these recommendations.
SCOPE AND METHODOLOGY
Scope
RIG/San Salvador conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions in accordance with our audit objective. We believe that the evidence obtained provides that reasonable basis. The purpose of the audit was to determine whether OTI is achieving its main goals of stabilizing Haiti through support for community revitalization, increased citizen engagement, and improved governance.
OTI awarded a cost-plus-fixed-fee completion form task order to Chemonics on March 14, 2011, to provide short-term technical assistance to reinforce stability and lay the groundwork for longer-term development in Haiti. As of May 3, 2012, OTI had obligated $46.5 million and disbursed $23 million. As of February 10, 2012, OTI had approved activities worth $22.9 million.
The audit was performed at USAID/Haiti and at the Chemonics project offices in Port-au-Prince, Cap-Haitien, and Saint-Marc from January 11 through April 11, 2012. The audit reviewed a judgmental sample of 22 ongoing and completed activities valued at $6.8 million from a universe of 141 activities approved, in process, or completed as of February 2012, valued at $22.9 million. The audit covered the period March 14, 2011, through February 10, 2012. We visited 19 sites in Port-au-Prince, Cap-Haitien, and Saint-Marc.
In planning and performing the audit, we assessed relevant internal controls that OTI used to manage its Haiti program. These controls included established policies and procedures for grant administration and program management, monitoring and evaluation processes, and reporting processes.
Methodology
To answer the audit objective, we interviewed OTI and USAID/Haiti officials, Chemonics employees, grantees, and beneficiaries. We reviewed and analyzed relevant documents including activity plans, grant agreements, grant and procurement documentation, activity progress and final evaluation reports, and environmental reviews. We judgmentally selected 22 ongoing and completed activities to obtain a sample of activities across regions, sectors, goals supported, and grantee type. We compared expected results with actual results by reviewing supporting documentation, observing activity implementation on site visits, and interviewing grantees and beneficiaries. Since the testing was based on a judgmental sample instead of a statistical one, the results and conclusions are limited to the items tested and could not be projected to the entire audit universe. To determine the reliability of computer-processed data in OTI’s activity database, for activities reviewed we compared grant data to source documents, and compared reported progress with conditions we observed during site visits.
MANAGEMENT COMMENTS
August 22, 2012
MEMORANDUM
TO: Regional Inspector General/San Salvador, Jon Chasson
FROM: USAID/DCHA/OTI Director, Robert Jenkins /s/
USAID/Haiti Acting Mission Director, Steve Olive /s/
SUBJECT: USAID Response to Audit of USAID’s Haiti Recovery Initiative Managed by the Office of Transition Initiatives (Report No. 1-521-12-XXX-P)
This memorandum transmits USAID’s response to the Regional Inspector General’s draft audit report “Audit of USAID’s Haiti Recovery Initiative Managed by the Office of Transition Initiatives,” dated July 26, 2012 for your consideration. USAID’s Office of Transition Initiatives (USAID/OTI) programs are, by nature, challenging to evaluate because of the iterative process applied to project development that is designed to work in rapidly changing program environments such as Haiti. We appreciate the efforts of the RIG staff to understand our unique programming approach and processes.
Regarding Recommendation 1: We agree with this recommendation. USAID/OTI is
currently in the process of engaging a local contractor to conduct third party monitoring and evaluation. One of the first deliverables of the contract will be the development of indicators that will be used for measuring and reporting against achievement of program goals for the remainder of the program. We will have these in place by December 31, 2012. It should be noted that it is challenging to attribute direct results in complex and fluid stabilization environments, and it is often the absence of destabilizing events that demonstrates stability in these historically volatile areas. Barring sudden major destabilization in our target communities, we anticipate that the indicators will offer qualitative and quantitative impact data and will be based on plausible correlation which may not be directly attributable to individual activities.
Regarding Recommendation 2: We agree with this recommendation. USAID/OTI
revised the “Activity Cycle and Guidebook” on May 2, 2012 to include guidance for the contractor on improving the implementation plans for each grant. The contractor will now submit basic implementation plans with the grant approval documentation and will submit detailed implementation plans with timelines for estimated completion of each portion of the activity no more than one week after the kick off meeting where the grantee signs the grant.
Regarding Recommendation 3: We agree with this recommendation. USAID/OTI will
work with the USAID/Haiti Program Office to complete the data quality assessments for the five indicators to which USAID/OTI contributes in the Mission’s Operation Plan and Performance Report. We will complete the data quality assessments no later than by December 1, 2012.
Regarding Recommendation 4 and 5: We agree with this recommendation.
Community engagement and ownership is a cornerstone of USAID/OTI programs worldwide and an integral part the USAID/OTI program, especially for our community stabilization activities. This has been an important aspect of our approach since the inception of the program; however, USAID/OTI recognizes that it has not been adequately documented and has faced particular challenges in activities that are not developed at the community level but which were requested by the Haitian national government or other USG actors. USAID/OTI revised the “Activity Cycle and Guidebook” on August 16, 2012, to specify that the contractor will include community engagement and communication as part of the implementation plans for community stabilization grants. The development section of the “Activity Cycle and Guidebook” also now explicitly states that community engagement will be conducted at the development stage of the activity design process to promote community ownership and sustainability for all community stabilization activities. The implementation section of the “Activity Cycle and Guidebook” also now explicitly states that community engagement will be conducted at the development stage of the activity design process to promote community ownership and sustainability for all community stabilization activities. The implementation section of the “Activity Cycle and Guidebook” also now states that the contractor submit an activity note for community stabilization activities describing the communication plan to increase the local population’s awareness of the activity and how it will benefit the community.
Regarding Recommendation 6: We agree with this recommendation.
Following the site visit by the audit team where the unmet expectations were raised by the grantee, USAID/OTI discussed the situation with our contractor and immediately began including check-in meetings with the grantee on the M&E plan for Short-Term Technical Assistance (STTA) activities that were in development. This included two with the same grantee of the audited activity, CNIGS. It should be noted that although there were communication issues with this STTA, for the majority of our other STTA activities to GOH entities, they have asked to extend the consultants because of how much they value the consultants’ contributions to the organization. USAID/OTI included grantee check-ins as an M&E requirement for STTA activities in the August 16, 2012 M&E guidebook.
Regarding Recommendation 7: We agree with this recommendation.
The activities reviewed under this audit were developed prior to Chemonics’ conducting an environmental compliance training in September 2011 in which USAID’s Regional Environmental Officer (REO) and the Mission Environmental Officer (MEO) participated. USAID/OTI has already completed its review of environmental documentation for current and planned activities, and worked with the MEO and REO to create three Umbrella Environmental Mitigation Plans and Reports (UEMPRs) for infrastructure, water and sanitation, and watershed management and irrigation. USAID/OTI creates individual EMPRs for all HRI activities that do not fall under one of the UEMPRs and now regularly reviews a spreadsheet that tracks the environmental compliance plans and regular reporting for all activities.
Regarding Recommendation 8: We agree with this recommendation.
On October 20, 2011, HRI added a section in the activity approval documents to document environmental compliance needs for each activity which must be completed before the activity is approved. In this section, the activity is classified based on its potential for environmental impact and what environmental documentation or mitigation works will be needed during implementation. Corresponding guidance was added to the “Activity Cycle and Guidebook” on May 2, 2012. Many of the common types of activities implemented by HRI that have potential for environmental impacts are implemented using an UEMPR. HRI’s efforts to improve environmental compliance
monitoring were so well-received that HRI was highlighted as an example and an HRI staff member spoke during the Environmental Compliance (Reg. 216) Workshop hosted by USAID/Haiti from April 23 - 27.
OFFICE OF INSPECTOR GENERAL
AUDIT OF USAID’S HAITI RECOVERY INITIATIVE ACTIVITIES MANAGED BY OFFICE OF TRANSITION INITIATIVES
AUDIT REPORT NO. 1-521-12-009-P SEPTEMBER 26, 2012
SUMMARY OF RESULTS
In January 2010 Haiti was struck by a devastating earthquake that leveled extensive portions of Port-au-Prince, killing or displacing hundreds of thousands of people. According to USAID’s Office of Transition Initiatives (OTI), the Haitian Government was largely incapacitated because significant numbers of government officials were killed and many buildings and records were destroyed.
In the immediate aftermath of the earthquake, OTI began implementing the Haiti Recovery Initiative (HRI). The initial focus was on supporting short- and medium-term activities aimed at stabilizing the Caribbean nation through assistance with community revitalization, improved governance, and economic strengthening.
In March 2011 OTI awarded Chemonics International Inc. a $53-million, 18-month contract to continue its work under the second phase of HRI (called HRI-II). HRI-II was designed to support the January 2011 Post-Earthquake U.S. Government Haiti Strategy; according to that, the United States would help Haiti strengthen its economy and public institutions in the three strategic development corridors of Port-au-Prince, Saint-Marc, and Cap-Haitien. As of May 3, 2012, OTI obligated $46.5 million and disbursed $23 million.
The Regional Inspector General/San Salvador (RIG/San Salvador) conducted this audit to determine whether HRI-II activities are achieving their main goals of stabilizing Haiti through community revitalization, increased citizen engagement, and improved governance.
The audit found that many OTI activities are providing benefits, such as:
OTI worked with the International Organization for Migration to relocate about 1,250 internally displaced families that were living in two camps in Petionville. The camps were cleared in December 2011.
OTI provided the Haitian Parliament with temporary offices and meeting space. Parliament’s headquarters partially collapsed during the earthquake. However, while individual activities had positive impacts locally, OTI was not using a structured system for measuring and reporting whether HRI-II as a whole was meeting its broader national objectives, and the lack of such a system made it difficult to measure the program’s impact.
Furthermore, HRI-II was not on track to complete all activities before the scheduled end date of September 2012. Budget line items in the contract provided for grants under contract ($37.3 million) and “non-grant under contract” activities, such as direct procurement of goods and services ($3.3 million) for a total of $40.6 million. As of February 2012, 141 activities worth about $22.9 million had been developed and approved, leaving $17.7 million available for new activities to be approved, implemented, completed, and closed in the 7 months left.
The audit also found some additional areas in which program management could be improved
as discussed in the following findings.
The monitoring and evaluation system was weak (page 3).
Community involvement in activities was not sufficient (page 7).
A short-term technical assistance activity did not meet expectations (page 7).
An environmental review for an activity was not performed and approved before
implementation (page 7).
This report recommends that OTI:
1. Implement a plan to measure and report progress toward meeting HRI-II’s goals for the remainder of the program (page 5).
2. Update its implementation plan for the remainder of the program to include estimated timelines for each phase of an activity for development, implementation, and closeout
(page 5).
3. Work with USAID/Haiti to conduct and document data quality assessments for the indicators OTI reports on for the mission’s operational plan and performance report (page 5).
4. Incorporate plans to engage communities when developing activities so the communities will sustain the efforts (page 6).
5. Implement communications plans into ongoing and new activities to make the local population aware of the activity and its benefits for the community (page 6).
6. Implement processes to assess short-term technical assistance with grantees throughout the grant cycle (page 7).
7. Review the existing environmental documentation of current and planned activities for compliance with environmental review requirements (page 9).
8. Require Chemonics to confirm that environmental reviews are completed before any activities begin (page 9).
Detailed findings appear in the following section. The audit scope and methodology are
described in Appendix I. Management comments are included in their entirety in Appendix II, and our evaluation of management comments is included on page 10 of the report.
AUDIT FINDINGS
Monitoring and Evaluation System Was Weak
USAID’s Automated Directives System (ADS) 200.6 defines performance management as the systematic process of monitoring the achievements of program operations; collecting and analyzing performance information to track progress toward goals; and using that information to influence how decisions are made and how resources are allocated. According to ADS 203.3.3.1, the performance management process has four principal steps: (1) establishing a performance management framework, (2) collecting and analyzing performance information, (3) using that information to influence decision-making and resource allocation throughout a program, and (4) communicating results. Missions use performance management plans as tools to plan and manage the process of monitoring, evaluating, and reporting progress toward achieving objectives. ADS does not provide a format for performance management plans; USAID offices and missions are allowed to use the format that best fits their needs.
OTI’s contract with Chemonics states, “Timely evaluation of activities and the program more broadly is a critical component of ongoing, updated analysis, which is accomplished through periodic strategic review sessions for regular evaluation of recent activities and the program.” Because OTI communicates frequently with its partners, it can adjust activities as needed throughout the course of a program. However, in the case of HRI-II, the lack of management tools to measure progress toward program goals made it difficult to measure the program’s impact. Specific monitoring and evaluation concerns are discussed below.
Program Management Was Not Comprehensive. HRI-II’s monitoring and evaluation focused primarily on the activity level and to a lesser extent on the program as a whole. Each activity had three levels of objectives and indicators to measure outputs, outcomes, and impact. HRI-II developed two guides, Activity Cycle and Guidebook and Activity Management Guide, to help employees manage the program. However, while the guides provided templates and addressed procedures for awarding and administering grants for the program, they did not contribute to monitoring the entire program.
According to the HRI-II contract, program monitoring efforts should include regular evaluations of the program, as well as the activities. These evaluations should allow the HRI-II staff to refine the program’s focus. The contract further states that the participants, nature, structure, and frequency of the evaluations depend on the country context, program needs, and the staff. In general, they should be conducted between two and four times a year.
OTI has conducted only one evaluation for HRI-II so far, and it took place in June 2011 to discuss and draft OTI’s strategy for Haiti, to build the new HRI-II team, and to propose ideas of where to conduct activities. OTI officials said they conduct quarterly “rolling assessments” to assess programmatic issues, program development, and implementation. In addition, OTI and Chemonics met weekly to cover program development and implementation, and to discuss problems, delays, and challenges.
While these meetings contribute to program development and implementation, they do not measure progress toward ultimate program goals. HRI-II Did Not Have Program-Wide Monitoring and Evaluation Plan. When the program started in March 2011, the staff developed a monitoring and evaluation guide that consisted of guidelines and procedures for developing monitoring and evaluation plans. However, it did not constitute a formal plan that HRI-II’s staff could use to track progress, nor did it explain how results would be measured and documented throughout the program.
Chemonics officials said they decided not to prepare a formal plan because the activities were so different from each other that monitoring and evaluation plans needed to be tailored to the activity level. At the time of the audit, HRI-II had 141 activities in progress. Because each activity had its own individual monitoring and evaluation plan, Chemonics was tracking 141 different plans, but had no single plan for the entire program. After an activity was completed, the Chemonics staff performed a final evaluation. OTI and Chemonics officials said they would use final evaluations, comparisons of “clusters” of similar activities, and “thematic reviews to provide lessons learned and a feedback loop” to guide future activities. However, given HRI-II’s short time frame and the different nature of the activities, this approach might not provide useful or timely information as a monitoring and evaluation tool.
Poorly Defined Indicators. USAID guidance states that performance indicators should be unambiguous about what is being measured and closely track the results they are intended to measure.
Some of the performance indicators Chemonics developed were not well-defined. One activity provided computer equipment, chalkboards, benches, chairs, desks, and school kits consisting of backpacks and school supplies to two public schools in target communities. Yet a performance indicator for that activity measured the number of students who returned to school. OTI developed the activity to help offset some of the costs borne by the parents in sending their children to school, but the performance measure did not correlate to the activity.
Other activities reviewed had only one or two indicators, and they were poorly defined; for example, one purportedly measured Number of government institutions supporting a conflict and/or fragility. An activity that provided a local municipality with an engineering study for improving one of the town’s main roads was measured by Number of reconstructed national governing institutions and systems that receive USG [U.S. Government] assistance to incorporate principles that support democracy and government legitimacy. None of these activities had indicators that clearly measured objectives.
Implementation Plans Lacked Timelines. OTI requires Chemonics to track activity data through OTI’s activity database, which includes a field for a detailed implementation plan. In addition, an activity or grant notes field provides chronological updates of the activity’s progress or anything discussed in implementation meetings. The combined information in the two fields helps OTI track an activity’s progress.
However, the detailed implementation plans for some of the activities reviewed did not have enough information for OTI to see whether the activity was on track to end on time and to meet its objectives. For example, the plan for providing Haiti’s Parliament with temporary offices and meeting space consisted of “The subcontractor will be responsible for the following:
1) Assembling and installation of steel-framed structures; 2) Connection of utilities.” No dates or estimated timelines were included.
Data Quality Assessments Not Performed. ADS 203.3.7.3 states that performance data reported to USAID/Washington in the performance plan and report must have had a data quality assessment (DQA) completed within the past 3 years. The goal of a DQA is to inform managers of data strengths and weaknesses and the extent to which data can be relied upon to influence management decisions.
Although results of OTI’s HRI-II activities were reported in USAID/Haiti’s fiscal year 2011 performance plan and report, the staff did not conduct DQAs of the data for the five indicators that it reported on in 2011.
All of the problems discussed in this finding stemmed from the fact that OTI did not make monitoring and evaluation a priority early in the HRI-II’s implementation. OTI officials said it started becoming a priority when Chemonics hired a full-time monitoring and evaluation officer in August 2011.
According to an OTI management review completed in January 2011, the HRI I program did not place enough emphasis on monitoring and evaluation. The team conducting the review recommended that OTI hire a permanent staff member to oversee monitoring and evaluation. OTI advertised for a foreign service national monitoring and evaluation specialist in April 2011, but officials said they did not receive any qualified applicants.
Under the program’s monitoring and evaluation framework, it was hard to consistently and effectively evaluate how results and targets were measured. This handicap could result in activity objectives not being met. To strengthen the program’s monitoring and evaluation system, we make the following recommendations
Recommendation 1. We recommend that USAID/Office of Transition Initiatives implement a plan to measure and report progress made toward achieving program goals for the remainder of the program.
Recommendation 2. We recommend that USAID/Office of Transition Initiatives require Chemonics International to update its implementation plans for the remainder of the program to include estimated timelines for each phase of an activity for development, implementation, and closeout.
Recommendation 3. We recommend that USAID/Office of Transition Initiatives
coordinate with USAID/Haiti to conduct and document results of data quality assessments for the indicators that USAID’s Office of Transition Initiatives reports on for the mission’s operational plan and performance report.
Communities Were Not Involved
Enough in Activities
As described in USAID’s sustainable development strategy, “Sustainable development mandates participation. It must involve, respond to, and be accountable to the people who will live with the results of the development effort.” According to the HRI-II contract, the goal of USAID/OTI’s efforts in Haiti is to help stabilize the country while laying the foundation for development and good governance. The contract further states that OTI would pursue all programming with an eye to community participation, government engagement, and alignment with the unified U.S. Government approach in Haiti.
HRI-II’s offices in Port-au-Prince, Cap-Haitien, and Saint-Marc have worked with local communities to develop activities tailored to meet their needs. Nonetheless, not all of the activities implemented in the corridor around Cap-Haitien were developed by the regional office that understood the social and economic context of the area, and not all activities implemented have involved community participation in a way that guarantees sustainability. OTI’s 2012 Management Review pointed out that the leadership of the Chemonics office in Port-auPrince often failed to communicate to the Cap-Haitien team the information necessary to conduct activities there that were developed or managed by the Port-au-Prince office. Program officials expressed concerns regarding activities planned by the Port-au-Prince office that did not utilize local knowledge. In addition, they said Chemonics used contractors from Port-auPrince to implement a number of activities in Cap-Haitien and Saint-Marc; these contractors brought their own people to do the jobs instead of hiring locals. As a result, residents saw jobs in their neighborhoods being done by outsiders, and without an understanding of the activities, they did not see how anyone local benefitted.
Program officials said that for some activities, particularly in the Cap-Haitien corridor, community members did not feel engaged and did not take ownership of the completed projects. Urban beautification projects in the towns of Limonade and Caracol failed for these reasons. The purpose of these projects was to improve public areas by installing plants and benches, as well as doing minor masonry work, and to project “a positive image of what role the nearby Caracol industrial park and other upcoming economic investments will play in citizens’ lives.” The landscaping work in Limonade was not completed as planned because of a lack of coordination between two HRI-II activities. Chemonics purchased and planted some seedlings for the town center, but they died from lack of care. According to the project’s final evaluation report, residents did not understand how the activity led to the beautification of the area nor did they associate it with the industrial park. Limonade’s mayor said the municipality could have been involved more in planning the activity to ensure its success.
Although many of OTI’s activities included community collaboration in development and implementation, some did not, and they were not very successful because OTI did not always incorporate plans into activity designs to promote community ownership and sustainability. Furthermore, OTI did not require Chemonics to implement a communication plan for each activity to increase the local population’s awareness of it and its benefits to the community. While OTI must strike a balance between coordinating closely with local communities and meeting its mandate of being fast and flexible, activities implemented with limited participation from local communities can hurt the prospects for sustainability. Therefore, we make the following recommendations.
Recommendation 4. We recommend that USAID/Office of Transition Initiatives incorporate plans to engage communities when developing activities so that they will take ownership and sustain the efforts.
Recommendation 5. We recommend that USAID/Office of Transition Initiatives require
Chemonics International to implement communications plans for ongoing and new activities to increase the local population’s awareness of the activities and how they will benefit the community.
Short-term Technical Assistance
Activity Did Not Meet Expectations
The audit team reviewed one of nine ongoing short-term technical assistance activities. These
provide an organization with an expert who trains or mentors employees for a short period of time. The HRI-II activity provided Haiti’s national mapping agency, Centre National de l’Information Geo Spatiale (CNIGS) with a geographic information system (GIS) specialist for 6 months because the agency lost most of its management staff, facilities, and equipment during the earthquake. The consultant was responsible for helping transfer mapping information from international humanitarian agencies to CNIGS and for working with other donor agencies in developing an information system for coordinating housing and neighborhood reconstruction. According to CNIGS and the consultant, the information system was not finished when her contract ended in February 2012. CNIGS’s acting director said she did not provide the help the agency needed and her input did not meet its needs. The acting director added that the consultant’s efforts to document user requirements for the information system were insufficient and would have to be redone by the system developer. The consultant said she had difficulties getting CNIGS staff members to discuss data sharing agreements, which delayed establishing the data-sharing protocols. In addition, the tasks of implementing the conceptual model of the information system and configuring the GIS server were only halfway completed by the end of her contract.
The activity did not meet its objectives because Chemonics did not communicate effectively with CNIGs or provide adequate support to the consultant. In October 2011, shortly after the consultant began working at the agency, she told Chemonics officials she was having difficulties discussing with CNIGS what they expected her to do in developing the system. Chemonics officials told her to continue to try to work with the agency.
Chemonics officials said they had separate meetings with the consultant and CNIGS monthly, but they focused on program development. The Chemonics officials said they never asked agency staff members whether the consultant was meeting their needs and learned only at the activity’s end that CNIGS was not satisfied with her. Short-term technical assistance might not help grantees if they do not get the type of assistance they need or if they do not use the expertise provided appropriately. To assist OTI in managing current and future short-term technical assistance grants, we make the following recommendation.
Recommendation 6. We recommend that USAID/Office of Transition Initiatives implement processes to assess short-term technical assistance with grantees periodically throughout the grant cycle.
Environmental Review Not Performed and Approved Before Implementation
ADS Chapter 204, “Environmental Procedures,” states that USAID must fully comply with environmental procedures established in the Code of Federal Regulations (22 CFR 216). USAID is required to “integrate environmental issues into its programs, both to meet U.S. Government legal environmental obligations and to optimize economic and social development results.”
Supplementary guidance to ADS 204 provides descriptions of environmental procedures to include in requests for proposals and award documents. The purpose is to give USAID time to conduct environmental reviews and incorporate environmental factors and mitigating measures in the design and approval of each program and activity before making an irreversible commitment of resources. ADS further states that the review should be done as early as possible in the design process to allow enough time for more detailed subsequent environmental review and concurrence, and for integrating environmental mitigations into the process.
Chemonics has established procedures for environmental reviews to screen each activity for potential adverse impacts, recommend determinations to categorize environmental risk, and identify mitigation and monitoring measures. The procedures for mitigation and monitoring include templates for environmental mitigation plans and reports; these were developed for USAID/Haiti to use when implementing infrastructure and related activities. Several of the activities the team reviewed were designated as “negative with conditions,” which applies to activities that could have moderate adverse environmental impacts, but could be managed effectively with appropriate mitigation and monitoring measures. For these activities, Chemonics used plans that included mitigation measures and reporting on monitoring of those measures.
In one of the activities, OTI did not complete the required environmental review and mitigation and monitoring plan before implementing the activity, which was designed in June 2011 to generate temporary employment opportunities through planting 700,000 jatropha seedlings in the Saint-Marc corridor. In Chemonics’s environmental review of the activity, it recommended a determination of “negative with conditions,” requiring an environmental mitigation and monitoring plan.
However, although the seedlings were planted in August 2011, OTI had not approved the environmental determination, and the mitigation and monitoring plan was not in effect. The environmental officer visited the jatropha plantation in August 2011 and said that no negative effects had occurred; the officer said Chemonics had done scientific research to be sure that the type of jatropha was proper for the area, but had not yet documented the environmental work performed. Subsequently, Chemonics developed a mitigation and monitoring plan and submitted it to USAID/Haiti and OTI for approval. The mission environmental officer and contracting officer’s representative approved the environmental mitigation plan and report in October and November 2011, respectively—after the jatropha had already been planted.
The lack of planning for environmental mitigations before implementation happened because OTI and Chemonics neglected to be sure that they complied with the requirements to complete the environmental documentation and to obtain the necessary USAID approvals before planting the trees. Potentially adverse environmental impacts can occur if proper mitigation and monitoring procedures are not put into place before implementing an activity and monitoring it. Therefore, we make the following recommendations.
Recommendation 7. We recommend that USAID/Office of Transition Initiatives review the existing environmental documentation of current and planned activities for compliance with environmental review requirements, and document the results of the review.
Recommendation 8. We recommend that USAID/Office of Transition Initiatives require Chemonics to complete environmental reviews before implementing activities.
EVALUATION OF MANAGEMENT
COMMENTS
Based on our evaluation of management comments on our draft report, we have determined that management decisions have been reached on all recommendations. In addition, final action has been taken on Recommendations 2, 4, 5, 6, 7, and 8. Our detailed evaluation of management comments follows.
Recommendation 1. OTI agreed to implement a plan to measure and report progress toward meeting program goals for the remainder of HRI-II. Officials said they would hire a local contractor to conduct third-party monitoring and evaluation. In addition, OTI plans to require the monitoring and evaluation contractor to develop indicators to be used for measuring and reporting on goals for the remainder of the program. OTI expects to have this in place by December 31, 2012. Based on OTI’s planned actions and estimated time frames, a management decision has been reached on this recommendation.
Recommendation 2. OTI agreed to update its implementation plan for the remainder of the program to include estimated timelines for each phase of an activity for development, implementation, and closeout. In May 2012 OTI updated its procedures and required Chemonics to submit detailed implementation plans that include timelines for estimated completion of each phase of an activity. Based on OTI’s response and supporting documents provided, final action has been taken on this recommendation.
Recommendation 3. OTI agreed to conduct and document DQAs of the indicators OTI reports on for the mission’s operational plan and performance report. In management comments, OTI officials said they would work with USAID/Haiti’s Program Office on the DQAs, and they expect to complete them by December 1, 2012. Based on OTI’s response and estimated time frames to complete the DQAs, a management decision has been reached on this recommendation.
Recommendations 4 and 5. To address the recommendations, OTI issued updated guidance to Chemonics in August 2012 that required it to incorporate community engagement and communication into each activity’s implementation plan. The plans should include information about how and when Chemonics plans to engage the communities in an activity and what communication tools it plans to use to increase activity impact by highlighting participation of local authorities and explaining the benefits to the community. OTI provided a copy of the updated guidance explicitly stating that community engagement is to be conducted in the activity development stage to promote community ownership and sustainability for all community stabilization activities. As a result of OTI’s response and completed actions, final action has been taken on these recommendations.
Recommendation 6. OTI agreed to implement processes to assess short-term technical assistance with grantees throughout the grant cycle. In response, OTI has implemented a procedure to include meetings with grantees as a monitoring and evaluation requirement for its short-term technical assistance activities. OTI provided copies of the updated monitoring and evaluation guidance issued to Chemonics that includes the new requirement. Based on OTI’s response and documented actions, final action has been taken on this recommendation.
Recommendations 7 and 8. OTI agreed to review the existing environmental documentation of current and planned activities for compliance with environmental review requirements, and to require Chemonics to confirm that environmental reviews are completed before any activities begin. OTI conducted a review of current and planned activities for environmental compliance. It also worked with USAID environmental officials to develop umbrella mitigation and monitoring plans for infrastructure, water and sanitation, and watershed activities. In addition, OTI added a step in its approval process to document environmental compliance requirements that must be completed before an activity can be approved. OTI provided documentation of the new requirement and a checklist documenting its environmental compliance review of the ongoing and planned activities. OTI also provided copies of the umbrella environmental mitigation and monitoring plans. Based on the actions taken by OTI and supporting documents, final action has been taken on these recommendations.
SCOPE AND METHODOLOGY
Scope
RIG/San Salvador conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions in accordance with our audit objective. We believe that the evidence obtained provides that reasonable basis. The purpose of the audit was to determine whether OTI is achieving its main goals of stabilizing Haiti through support for community revitalization, increased citizen engagement, and improved governance.
OTI awarded a cost-plus-fixed-fee completion form task order to Chemonics on March 14, 2011, to provide short-term technical assistance to reinforce stability and lay the groundwork for longer-term development in Haiti. As of May 3, 2012, OTI had obligated $46.5 million and disbursed $23 million. As of February 10, 2012, OTI had approved activities worth $22.9 million.
The audit was performed at USAID/Haiti and at the Chemonics project offices in Port-au-Prince, Cap-Haitien, and Saint-Marc from January 11 through April 11, 2012. The audit reviewed a judgmental sample of 22 ongoing and completed activities valued at $6.8 million from a universe of 141 activities approved, in process, or completed as of February 2012, valued at $22.9 million. The audit covered the period March 14, 2011, through February 10, 2012. We visited 19 sites in Port-au-Prince, Cap-Haitien, and Saint-Marc.
In planning and performing the audit, we assessed relevant internal controls that OTI used to manage its Haiti program. These controls included established policies and procedures for grant administration and program management, monitoring and evaluation processes, and reporting processes.
Methodology
To answer the audit objective, we interviewed OTI and USAID/Haiti officials, Chemonics employees, grantees, and beneficiaries. We reviewed and analyzed relevant documents including activity plans, grant agreements, grant and procurement documentation, activity progress and final evaluation reports, and environmental reviews. We judgmentally selected 22 ongoing and completed activities to obtain a sample of activities across regions, sectors, goals supported, and grantee type. We compared expected results with actual results by reviewing supporting documentation, observing activity implementation on site visits, and interviewing grantees and beneficiaries. Since the testing was based on a judgmental sample instead of a statistical one, the results and conclusions are limited to the items tested and could not be projected to the entire audit universe. To determine the reliability of computer-processed data in OTI’s activity database, for activities reviewed we compared grant data to source documents, and compared reported progress with conditions we observed during site visits.
MANAGEMENT COMMENTS
August 22, 2012
MEMORANDUM
TO: Regional Inspector General/San Salvador, Jon Chasson
FROM: USAID/DCHA/OTI Director, Robert Jenkins /s/
USAID/Haiti Acting Mission Director, Steve Olive /s/
SUBJECT: USAID Response to Audit of USAID’s Haiti Recovery Initiative Managed by the Office of Transition Initiatives (Report No. 1-521-12-XXX-P)
This memorandum transmits USAID’s response to the Regional Inspector General’s draft audit report “Audit of USAID’s Haiti Recovery Initiative Managed by the Office of Transition Initiatives,” dated July 26, 2012 for your consideration. USAID’s Office of Transition Initiatives (USAID/OTI) programs are, by nature, challenging to evaluate because of the iterative process applied to project development that is designed to work in rapidly changing program environments such as Haiti. We appreciate the efforts of the RIG staff to understand our unique programming approach and processes.
Regarding Recommendation 1: We agree with this recommendation. USAID/OTI is
currently in the process of engaging a local contractor to conduct third party monitoring and evaluation. One of the first deliverables of the contract will be the development of indicators that will be used for measuring and reporting against achievement of program goals for the remainder of the program. We will have these in place by December 31, 2012. It should be noted that it is challenging to attribute direct results in complex and fluid stabilization environments, and it is often the absence of destabilizing events that demonstrates stability in these historically volatile areas. Barring sudden major destabilization in our target communities, we anticipate that the indicators will offer qualitative and quantitative impact data and will be based on plausible correlation which may not be directly attributable to individual activities.
Regarding Recommendation 2: We agree with this recommendation. USAID/OTI
revised the “Activity Cycle and Guidebook” on May 2, 2012 to include guidance for the contractor on improving the implementation plans for each grant. The contractor will now submit basic implementation plans with the grant approval documentation and will submit detailed implementation plans with timelines for estimated completion of each portion of the activity no more than one week after the kick off meeting where the grantee signs the grant.
Regarding Recommendation 3: We agree with this recommendation. USAID/OTI will
work with the USAID/Haiti Program Office to complete the data quality assessments for the five indicators to which USAID/OTI contributes in the Mission’s Operation Plan and Performance Report. We will complete the data quality assessments no later than by December 1, 2012.
Regarding Recommendation 4 and 5: We agree with this recommendation.
Community engagement and ownership is a cornerstone of USAID/OTI programs worldwide and an integral part the USAID/OTI program, especially for our community stabilization activities. This has been an important aspect of our approach since the inception of the program; however, USAID/OTI recognizes that it has not been adequately documented and has faced particular challenges in activities that are not developed at the community level but which were requested by the Haitian national government or other USG actors. USAID/OTI revised the “Activity Cycle and Guidebook” on August 16, 2012, to specify that the contractor will include community engagement and communication as part of the implementation plans for community stabilization grants. The development section of the “Activity Cycle and Guidebook” also now explicitly states that community engagement will be conducted at the development stage of the activity design process to promote community ownership and sustainability for all community stabilization activities. The implementation section of the “Activity Cycle and Guidebook” also now explicitly states that community engagement will be conducted at the development stage of the activity design process to promote community ownership and sustainability for all community stabilization activities. The implementation section of the “Activity Cycle and Guidebook” also now states that the contractor submit an activity note for community stabilization activities describing the communication plan to increase the local population’s awareness of the activity and how it will benefit the community.
Regarding Recommendation 6: We agree with this recommendation.
Following the site visit by the audit team where the unmet expectations were raised by the grantee, USAID/OTI discussed the situation with our contractor and immediately began including check-in meetings with the grantee on the M&E plan for Short-Term Technical Assistance (STTA) activities that were in development. This included two with the same grantee of the audited activity, CNIGS. It should be noted that although there were communication issues with this STTA, for the majority of our other STTA activities to GOH entities, they have asked to extend the consultants because of how much they value the consultants’ contributions to the organization. USAID/OTI included grantee check-ins as an M&E requirement for STTA activities in the August 16, 2012 M&E guidebook.
Regarding Recommendation 7: We agree with this recommendation.
The activities reviewed under this audit were developed prior to Chemonics’ conducting an environmental compliance training in September 2011 in which USAID’s Regional Environmental Officer (REO) and the Mission Environmental Officer (MEO) participated. USAID/OTI has already completed its review of environmental documentation for current and planned activities, and worked with the MEO and REO to create three Umbrella Environmental Mitigation Plans and Reports (UEMPRs) for infrastructure, water and sanitation, and watershed management and irrigation. USAID/OTI creates individual EMPRs for all HRI activities that do not fall under one of the UEMPRs and now regularly reviews a spreadsheet that tracks the environmental compliance plans and regular reporting for all activities.
Regarding Recommendation 8: We agree with this recommendation.
On October 20, 2011, HRI added a section in the activity approval documents to document environmental compliance needs for each activity which must be completed before the activity is approved. In this section, the activity is classified based on its potential for environmental impact and what environmental documentation or mitigation works will be needed during implementation. Corresponding guidance was added to the “Activity Cycle and Guidebook” on May 2, 2012. Many of the common types of activities implemented by HRI that have potential for environmental impacts are implemented using an UEMPR. HRI’s efforts to improve environmental compliance
monitoring were so well-received that HRI was highlighted as an example and an HRI staff member spoke during the Environmental Compliance (Reg. 216) Workshop hosted by USAID/Haiti from April 23 - 27.
-3.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
