PID / DataSource /000020051-20140625234238.txt
gkim93's picture
Upload all the data source files
cf652fa
 ICRR 14105
Report Number : ICRR14105
IEG ICR Review
Independent Evaluation Group
1. Project Data: Date Posted : 10/03/2013
Country : Mongolia
Project ID : P099321 Appraisal Actual
Project Name : Renewable Energy US$M ):
Project Costs (US$M): 23.0 22.23
For Rural Access
Project (reap)
L/C Number : CH263 Loan/ US$M):
Loan /Credit (US$M): 3.50 3.43
Sector Board : Energy and Mining Cofinancing (US$M):
US$M ): 9.50 8.81
Cofinanciers : Government of Board Approval Date : 12/19/2006
Netherlands, GEF Closing Date : 12/31/2011 06/30/2012
Sector (s): Renewable energy (90%); Central government administration (8%); Health (1%); General
education sector (1%)
Theme (s): Rural services and infrastructure (50% - P); Regulation and competition policy (25% - S);
Climate change (25% - S)
Prepared by : Reviewed by : ICR Review Group :
Coordinator :
Varadarajan Atur Robert Mark Lacey Soniya Carvalho IEGPS1
2. Project Objectives and Components:
a. Objectives:
As per the project appraisal document (PAD, pp 4-5), the project development objective (PDO) is “to increase
access to electricity and improve reliability and affordability of electricity services among the herder population and in
off-grid Soum-centers�? (a Soum is a district within a province) and the global environment objective (GEO) is “to
remove barriers to the development and use of renewable energy technologies in grid and off-grid connected
systems and reduce emissions of carbon dioxide.�?
According to the Financing Agreement (p 5), “the objective of the Project is to assist the Recipient in: (i) expanding
and improving access to electricity and reliability of electricity services in selected off-grid Soum centers and among
herder population; and (ii) removing barriers to the scale-up of renewable energy use.�? This is identical to the PDO
stated in the GEF Grant Agreement (p6). The PDO from the Financing Agreement will be used as the basis for this
evaluation.
b.Were the project objectives/key associated outcome targets revised during implementation?
No
c. Components:
There were three components:
Component 1: Herders’ Electricity Access (US$11.6 m at appraisal; US$11.75 m at completion) involved two
subcomponents: (i) the investment cofinancing subcomponent (US$10.7 m at appraisal; US$11.0 m after
reallocation and at completion) supported a combined 67,224 units (50,000 targeted at appraisal) of solar home
systems (SHS) and small wind turbine systems (WTS) for the Herder population through smart subsidies (i.e.
targeted subsidies to improve effectiveness) to buy down the investment cost for beneficiaries. This was financed by
the Government and the grant from the Netherlands; and (ii) the complementary TA part (US$0.9 m at appraisal;
US$0.76 m at completion) supported sales / service network development, equipment quality standards & control
and marketing and sales/service.
Component 2: Soum Center Electricity Service (US$10.1 m at appraisal; US$9.31 m at completion) involved two
subcomponents: (i) investment financing for rehabilitation of mini grids in about 30 off-grid Soum Centers (SCs)
(US$0.9 m at appraisal; US$0.88 m at completion) and conversion of existing diesel generation units to renewable or
renewable-diesel hybrid systems (R/RDHS) in about 20 of the rehabilitated SCs (US$8.3 m at appraisal; US$8.0 m
at completion); and (ii) complementary technical assistance to support policy, regulation, energy management and
capacity building in SCs, and feasibility studies for hybrid systems (US$0.89 m at appraisal; US$0.43 m at
completion).
Component 3: National Capacity Building (US$1.31 m at appraisal; US$1.17 m at completion) aimed at national
renewable energy policy and regulatory framework development and strengthening National Renewable Energy
Center (NREC) in project management, monitoring, evaluation activities. Some of these activities were also directed
towards removing barriers to renewable energy use.
A restructuring approved in August-2009 reallocated savings to modestly increase the allocation to the Herders'
Access component. A second restructuring approved in November-2011 extended the closing date by six months.
d. Comments on Project Cost, Financing, Borrower Contribution, and Dates:
Project cost. The final project of US$22.23 m is close to the appraisal estimate of US$23.0 m and the small
savings allowed expanding the access component. Considering that most of the components were substantially
implemented as planned, the project costs appear realistic for the nature and scope of activities implemented in the
local conditions. However, it is not clear from the PAD or ICR how the costs borne by the herders are accounted for
in the projects costs; it is good practice to include these costs in the project cost and financing plan.
Financing. The financing plan included an IDA Grant (US$3.50 m; actual US$3.43 m), GEF Grant (US$3.50 m;
actual US$2.46 m), a Grant from the Government of the Netherlands (US$6.0 m; actual US$5.85 m) and the
contribution of the Government of Mongolia (US$10.0 m; actual US$10.0 m). The “smart subsidy�? part was fully
covered out of the contributions from the Netherlands and Mongolia. As noted above, for the SHSs under component
1, the financing provided by herders themselves for the portion of costs above the subsidy part should have been
clearly shown.
Borrower Contribution. The Borrower’s contribution was substantial at about 44% of project costs and financed
about 63% of the smart subsidy component.
Dates. Even though the solar home system (SHS) component took two years initially to take off, the project required
only a six-month extension to complete all subcomponents. The ICR attributes the initial delay to the piloting of the
mechanism relying on private dealers to self-finance the SHSs first and then to recover costs from the project funds
and from buyers.
3. Relevance of Objectives & Design:
a. Relevance of Objectives:
The project objectives are relevant to Mongolia’s scattered rural population, including many with nomadic life style.
The project objectives are equally relevant to the Bank's priorities in Mongolia, as noted in the Country Partnership
Strategy for Fiscal Years 2013-2017 (p. 33). The necessity for rural electrification was also noted in the Bank’s
Country Assistance Strategy (CAS) of 2005-08. The Government’s own emphasis on rural electrification and large
budgetary allocation, including for this project, underlines the pertinence of the project’s objectives. The objectives
are in line with the Bank’s Strategic Framework on Development and Climate Change.
Relevance of the objectives is rated high
b. Relevance of Design:
The components were designed to support the achievement of the project development and Global Environment
objectives, and enabled synergy in the use of IDA, GEF and other funds in the financing plan. The three components
aimed at the particular target beneficiary group, namely, nomadic herders through component 1, Soum centers
through component 2 and national frameworks through component 3, with component specific technical assistance
embedded in them. The choice of portable solar and wind systems was appropriate for the mobile herder population,
as was the renewable-diesel hybrid for Soum centers. The design also included proven methods to motivate the
target population to adopt the new technologies.
Relevance of Design is rated high
4. Achievement of Objectives (Efficacy):
Project Development Objective
(i) to expand and improve access to electricity among herder population
Outputs: The project enabled distribution and sales of 67,224 SHS systems to herder populations compared to
66,816 units anticipated from the project component at the time of restructuring / reallocation of some savings. The
ICR notes that these systems covered up to 62.5% of the herder population in 331 Soums, in line with the expanded
reach (compared to 50% from 50,000 units estimated at appraisal).
Outcomes: The ICR records the favorable results from a qualitative survey of herders covering 10 Soums, wherein
beneficiaries expressed a high degree of satisfaction with improved lighting and access to communication, and
information. They also demonstrated a favorable attitude towards the new systems (Annex 5). A typical herder
family’s annual savings on lighting alone is estimated at $35, which would be about 5% of median income (the
survey put no monetary value on the other benefits indicated).
Efficacy of this objective is rated “high�?.
(ii) to expand and improve access to electricity in selected off-grid Soum centers
Outputs: The project’s reach under this component was significantly smaller than was estimated at appraisal.
Rehabilitation of mini grids was performed only at 15 instead of 30 Soum centers; installation of new technology
equipment was made in 15 SCs instead of 20 planned, of which 11 of were funded by Government itself; further, all
these 15 SCs with new systems were transferred to Aimag (equivalent of province) utilities and hence were not
under the management of Soum centers by project closure; the project set up only 15 Soum user association
compared to 30 planned at appraisal. The ICR notes that the Government allocated substantial resources (about
US$51 million) for rural electrification initiatives, but is not clear what part of the results achieved is due to this
government initiative. Therefore, the claim that the project benefitted 18,410 people in off-grid SCs compared to
16,000 at appraisal is not entirely attributable to the Bank project; it is also not clear if it can be linearly prorated
since Soum centers would have different population bases.
Outcomes: The ICR records the results from a survey (termed rapid appraisal) conducted in October 2012 (Annex 5
of the ICR), after the formal closing of the project. The survey covered 3 sites in two regions and included both
Government financed and Bank financed systems and involved 34 interviews. Generally, the survey found the
following: (i) the new services are cost effective where recipients could use modern and energy efficient lamps; (ii)
residents indicated increased comfort, leisure, improved access to communication, ability to store food and thus the
ability to save money; and (iii) availability of medical care from equipment using electricity and some perceived
growth in small business like bakery, workshop and the like. The sample size was, however, limited, and a broader
survey would be needed to confirming these findings. Efficacy of this objective is rated “modest�?.
(iii) to improve reliability of services to herder population For the herder population, the replacement of candles and
kerosene lamps with SHSs is an immensely superior choice, on the basis of lumens (measure of quality and quantity
of light), safety, and environmental protection. Since the SHSs also had storage capacity, the herders could use
them any time when needed with ease and comfort. These factors could be regarded as proxies of reliability with
safety and comfort. Efficacy of this objective is rated “high�?.
(iv) to improve reliability of electricity services to selected off-grid Soum centers The ICR does not provide any
evidence of improved reliability to off-grid Soum Centers (e.g. reduced outages, grid losses, etc); also while it is not
clear if the 4 solar-diesel hybrids installed per se improved reliability due to choice of technology, the rest of the
Soum Centers with wind-diesel hybrids are reported to have encountered operational problems at close of project (p
34 of ICR). The survey does not indicate the proper baseline to correctly attribute and assess the results conveyed
therein. Efficacy of this objective is rated “negligible�?.
v) to remove barriers to the scale-up of renewable energy use. The barriers confronting the implementation of the
two investment components were collectively overcome by the project’s technical assistance support as well as by
the Government’s own initiatives (for example, passing the Renewable Energy Law in 2007; converting the National
Renewable Energy Center (NREC) to a self-financing structure in 2009; substantial allocation of funds for rural
electrification). Experience from other countries has demonstrated that these interventions are necessary for
overcoming initial barriers, as this project has also shown. However, further expansion and sustainable use of
renewable energy in the Soum Centers and among herder populations need to be monitored and the ICR does not
provide specific evidence of such monitoring taking place or a process in place to do so. The market for such
renewable energy adoption is also not significant (about 3% according to the PAD). Since the effectiveness of the
after sales service and battery recycling is still to be verified, it can only be argued that the direction of efforts are
appropriate but that the results are yet to be seen. Efficacy of this objective is rated “modest.�?
Global Environment Objective (Unrated)
(vi) to reduce emissions of carbon dioxide. The SHS systems sold under the project replaced kerosene in the herder
homes and the solar/wind hybrids with diesel in Soum centers reduced use of diesel correspondingly as noted by the
ICR (Annex 3). The ICR estimated the equivalent reductions in carbon dioxide to be about 11,333 tons per year
(Annex 3 of ICR); based on the 67,224 SHS systems sold, the carbon dioxide reduction would be about 8,390 tons
per year (using assumptions per Annex 16 of PAD), which imputes 2,943 tons per year reductions due to 15 hybrids
at 15 Soum centers installed under the project. This is in line with the appraisal estimates of 9,000 tons per year
(about 6,240 tons from SHSs and 2,760 tons from hybrids).
5. Efficiency:
The project uses a cost effectiveness approach aimed at identifying the amount of capital subsidy necessary for (1)
the herder population to realize perceptible savings from their current expenditure on candles and kerosene of about
US$86 per annum based on substitution; and (2) the SCs to fully recover costs over the affordable tariffs of
US$0.17/kWh, using a discount rate of 12%.
For the GEO, the project aims to show carbon dioxide (CO2) reductions through avoided use of kerosene and diesel.
SHSs for herder population: The proposed subsidy of $80 per SHS at appraisal was based on costs of typical
systems (about $160 for <50 Watt peak [50Wp]), baseline studies in Mongolia, and benchmarks in other countries.
The subsidy led to net savings of about US$44 per annum (just over 50% of the subsidy). The SHSs provided nearly
15 times the amount of quality light than would be available from candles and kerosene (measured as lumen-hours
per year) for standard use. The actual costs of SHSs increased nearly 150% due to choice of larger than 50Wp
systems and necessitated doubling the subsidy to $160 for herder families. Because the costs of candles and
kerosene (alternatives) also went up (to $124 compared with $86 at appraisal), the savings to families at project
closure was estimated at $35 instead of $44 at appraisal.
Soum centers: At appraisal, several types of replacement alternatives for hybrids were considered and shown to be
efficient choices at 12% discount rate (Annex 9 of PAD). The ICR based its estimates on the solar-diesel hybrid due
to ease and quality of data available at completion (Annex 3 of ICR). Due to the rapidly decreasing trend in costs of
solar panels (about 33% since 2009 - see p34), the ICR duly notes the favorable impact on Soum centers and users
in sustaining the choice of such systems for expansion and replacement.
Global environmental benefits: The ICR does not provide details of actual measurements or good proxy estimates
for avoided use of kerosene and diesel and also does not adjust the baseline estimates due to expanded scope of
the herder component following reallocation of funds at restructuring. This notwithstanding, the penetration of SHSs
and wind/solar hybrids in Soum centers in the scale achieved by the project attests to significant avoidance of
kerosene and diesel and the CO2 estimates could be surmised to be in the vicinity of adjusted baseline estimates or
even slightly lower due to the corresponding smaller number of SCs covered under the project.
There were no significant operational or administrative inefficiencies.
Overall efficiency is rated “substantial�?.
ERR )/Financial Rate of Return (FRR)
a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the
re-
re -estimated value at evaluation :
Rate Available? Point Value Coverage/Scope*
Appraisal No
ICR estimate No
* Refers to percent of total project cost for which ERR/FRR was calculated.
6. Outcome:
The project objectives were highly relevant as was the design, including the approach, the components and the
clarity of the results framework. Planned outputs were achieved for the most part, and the outcomes demonstrated
the technical and financial feasibility of the options, and indicated overcoming of barriers to adoption of new and
small scale renewable energy technologies. The specific parts of the objectives were achieved in a mixed manner
(as shown by varied ratings from high to negligible for outcomes). Considering that the efficacy of the Soum center
component, with 42% of funding, is modest to negligible, and the efficiency rating is substantial, the overall outcome
is rated as moderately satisfactory.
a. Outcome Rating : Moderately Satisfactory
7. Rationale for Risk to Development Outcome Rating:
At project completion, (i) the SHSs were well accepted by the herder population, which was also evidenced by the
large penetration supported by the parallel Government initiatives; (ii) the solar hybrids with diesel in select Soum
centers had demonstrated operability, while the wind hybrids encountered some technical issues; and (iii) the NREC
had a self-financing structure with enabling legal and regulatory frameworks in place. The operating issues of wind
hybrids need to be addressed. Also, the safe disposal and handling of old batteries needs care and attention. The
country’s overall growth and rapid income gains should help sustain Government’s continuing interest and focus on
rural development priorities.
a. Risk to Development Outcome Rating : Moderate
8. Assessment of Bank Performance:
a. Quality at entry:
The Bank ensured consistency of project objectives with Government priorities and with the Country
Partnership Strategy and appropriately considered the alternatives in the design of components and smart
subsidies. It pooled IDA, GEF and other bilateral grants to support the the removal of barriers to the adoption of
new technologies. The Bank also promoted international standards for new products in renewable technologies,
an innovative approach to "crowding in" of the private sector (though it was over ambitious on hind sight
regarding the sector's financial capacity) and the use of technologies that were already proven in other countries.
The component-specific technical assistance, along with support aimed at regional and national levels, were
appropriate for the project. The risks were properly identified during preparation, as were the mitigation
measures adopted. M&E design was adequate.
at -Entry Rating :
Quality -at- Satisfactory
b. Quality of supervision:
As the ICR notes, the Bank’s supervision was more vigorous on investment components and less so on the
technical assistance activities, which was borne out in the canceled parts of the latter, especially unused GEF
funds. The Bank team seems to have taken a long time in the initial phase (first two years) before modifying the
procurement approach. It is not clear if the decision to adopt a bulk purchase alternative using the project funds
could have been made sooner, say six months to a year, thereby completing the project on or before the original
schedule. Nevertheless, the six month extension for completing the scope of the project seems reasonable given
the innovative features of the project and the relatively low capacity of the institutions involved at the beginning of
implementation. The focus on the SC component could have been better as pointed in the gaps of information
gathered to evidence reliability improvements as well as concerning the technical problems in wind hybrids
adopted in certain Soum centers. The efforts to conduct surveys and rapid appraisal are noteworthy and could
be followed up for monitoring sustained use and spread of adoption of SHS by herder population.
Quality of Supervision Rating : Moderately Satisfactory
Overall Bank Performance Rating : Moderately Satisfactory
9. Assessment of Borrower Performance:
a. Government Performance:
The ICR recognizes the Government’s role, commitment, funding, and performance. Its rapid responsiveness
to certain issues – for example, enhancing the subsidy to the herder populations – helped recover the time lost
during the initial sluggish start of the project. The Government’s sustained commitment to rural electrification
overall, including through large budget outlays, helped the momentum for the Bank’s parallel project in a
synergistic way. The timely adoption of supporting regulations by the National Renewable Energy Center (NREC)
and renewable energy law are testimony to the Government’s ownership and performance.
Government Performance Rating Satisfactory
b. Implementing Agency Performance:
Considering the innovative aspects of the project, the Project Implementation Unit (PIU) in the NREC seems
to have performed quite well, adjusting to new demands as they arose (e.g. the change in procurement method,
contract difficulties experienced). The project had built in technical assistance for such capacity building at the
outset. The rapid pickup of the implementation pace after the initial lull demonstrates the PIU’s sustained efforts
and focus on the project’s objectives, including the surveys undertaken. However, the ICR also candidly points
out the shortcomings of the PIU during implementation, especially regarding record keeping for project
management and fiduciary purposes. The PIU seems to have addressed the documentation concerns after
receiving training (pp 20-21 of ICR).
Implementing Agency Performance Rating : Moderately Satisfactory
Overall Borrower Performance Rating : Moderately Satisfactory
10. M&E Design, Implementation, & Utilization:
a. M&E Design:
The M&E framework was designed for the components to track the outputs and outcomes and the achievement of
the project development and Global Environmental objectives. The results framework identified and established the
outcome indicators by component and over time in the implementation plan. The survey approach was appropriate
to track the mobile herder population’s adoption of SHS systems, use, etc., and covered 10 Soum centers. A similar
survey of Soum centers for monitoring the benefits of the electricity service covered only 3 Soum centers. The PIU
was responsible for M&E and baseline data and targets were defined for most of the indicators.
b. M&E Implementation:
Even though the PIU was provided with training in M&E at the launch of the project, the ICR notes the difficulties
encountered by the PIU, and hence the Bank team, in assessing progress in some key areas during implementation.
The tracking of key data for assessing the GEO achievement could be done only at the end of the project due to
specific information needed on Soum centers. While reasonably good estimates can be made of the herder
component based on baseline and average displacement for kerosene, more pertinent data would have been
needed for properly estimating the avoided diesel use at Soum centers with hybrid systems.
c. M&E Utilization:
The surveys and rapid appraisal provide an important basis for improving the quality of data to estimate the
benefits and subsequently to fine tune the approach in the continuing rural electrification initiatives. The efforts made
by the PIU and the Bank team to disseminate the results and lessons from the project are noteworthy and attest to
good practice.
M&E Quality Rating : Substantial
11. Other Issues
a. Safeguards:
This is a category C project for safeguards compliance and management. The project had positive benefits for
environment. The ICR candidly notes the shortcoming and oversight in the failure to tackle early on the potential
issues with handling and disposal of used batteries for SHS systems. Even though the individual SHS system battery
is small, the widespread use by a large number of herders does pose health hazards if the batteries are not properly
handled and disposed after their useful life span. The team appropriately supported a study to examine the feasibility
of a battery recycling facility, and the ICR records the interest already shown by the private sector in this regard
(page 15). It is important to follow up on the handling nand disposal of used batteries in view of their potential
adverse impacts. The project team confirmed that the project was in compliance with the applicable safeguards
policies of the Bank at all times.
b. Fiduciary Compliance:
Both procurement and financial management are noted as in compliance with agreed standards, except for minor
shortcomings in records maintenance for a short time during implementation. The ICR reports that the shortcomings
were handled by the PIU and the Bank team in a timely manner, and the project team confirmed that the audit
reports were satisfactory (that is, unqualified) throughout implementation. Delays in procurement were handled by
changing the approach for bulk procurement of SHSs and a contract dispute was able to be settled in local courts.
c. Unintended Impacts (positive or negative):
d. Other:
12.
12. Ratings : ICR IEG Review Reason for
Disagreement /Comments
Outcome : Satisfactory Moderately The outcomes planned for the Soum
Satisfactory center component were not achieved
fully and no evidence was provided to
support improvements in reliability. The
effectiveness of technical assistance
for barrier removal and safe handling of
used batteries was deficient .
Risk to Development Moderate Moderate
Outcome :
Bank Performance : Satisfactory Moderately As the ICR itself notes, there were
Satisfactory shortcomings in the responsiveness of
Bank supervision to handle issues in a
timely manner
Borrower Performance : Satisfactory Moderately As the ICR itself notes, the PIU
Satisfactory performance regarding record keeping
for fiduciary aspects was less than fully
satisfactory.
Quality of ICR : Satisfactory
NOTES:
NOTES
- When insufficient information is provided by the Bank
for IEG to arrive at a clear rating, IEG will downgrade
the relevant ratings as warranted beginning July 1,
2006.
- The "Reason for Disagreement/Comments" column
could cross-reference other sections of the ICR
Review, as appropriate.
13. Lessons:
The lessons of broader significance from the ICR are the following:
(i) Transfer of international experience requires care and adaptation: International experiences in energy
efficiency promotion are generally transferable, but to be effective they have to be carefully selected and
customized to suit local conditions.
(ii) Smart subsidies are effective: Targeting subsidies are effective in promotion of, and removal of barriers to,
energy efficiency. The effectiveness of such subsidies would be better if distortions in fuel subsidies are minimized
or removed (e.g. diesel subsidies to utilities would reduce incentives to select other renewable options).
(iii) Safe handling and disposal of batteries is essential: In addition to the above two lessons noted in the ICR,
the project has highlighted the need for adopting appropriate safety and mitigation measures for ensuring proper
handling and disposal of batteries with hazardous chemicals, including adequate funding and institutional
responsibility for ensuring implementation.
14. Assessment Recommended? Yes No
Why? The project offers important lessons regarding the choice approach, technologies, etc., for off-grid and
mini-grid development in rural electrification and access, which are important challenges in the power sector at this
time. The lessons could be beneficial, for example in the South Asia and Africa Regions, both of which have large
populations without electricity access.
15. Comments on Quality of ICR:
The ICR is comprehensive, candid and provides evidence for the most part of assessment. It provides sufficient
information to validate the outputs and outcomes and the quality of analysis is satisfactory. The format is in
conformity with guidelines and the lessons are useful and relevant for practical operational purposes. In Annex 5, the
various survey responses could be tabulated for better understanding and clarity. The ICR could also have provided
more details of the Government funded wind-diesel hybrids.
a.Quality of ICR Rating : Satisfactory