Transforming the energy efficiency market in California: Key findings, lessons learned and future directions from California's market effects studies
Introduction
In an October 2007 decision (D.07-10-032), the CPUC directed its staff to explore (during 2008–2009) the ability to credibly quantify and credit “nonparticipant spillover” market effects, and to report on the ability of current protocols to measure nonparticipant spillover savings for the 2006–2008 program cycle. The market effects evaluation protocol provides the following definition of market effects (California Public Utilities Commission (CPUC), 2006):
A change in the structure of a market or the behavior of participants in a market that is reflective of an increase in the adoption of energy-efficient products, services, or practices and is causally related to market interventions…” where a “market” is defined as “the commercial activity (manufacturing, distributing, buying and selling) associated with products and services that affect energy usage.
In the October 2007 decision, the CPUC directed its staff to report its findings following the process evaluation and market impact studies of the 2006–2008 program cycle on the ability of current protocols to measure such “nonparticipant spillover” savings and to propose possible revisions to market effects protocols, utility savings goals, or performance incentive mechanisms for subsequent action by the CPUC. Consequently, the CPUC decided to examine possible market effects in CFLs, RNC, and HBL (referred to as the “market effects studies”). Working with the CPUC, CIEE developed study plans for, and assisted in overseeing, each of these market effect studies.2
The market effects studies had three primary objectives (Vine, 2011):3
- •
Understand the cumulative effects of California's energy-efficiency programs on the target market.
- •
Quantify 2006–2008 kW-h and kilowatt savings (if any) caused by the above potential market effects and not claimed as direct or participant spillover savings.
- •
Support the CPUC's strategic planning efforts by clarifying whether savings from potential market effects can be quantified with sufficient reliability to be treated as a resource.4
Section snippets
Overview of the market effects studies5
As shown in Table 1, each of the studies addressed the above objectives with evaluation methodologies relying on a diverse set of data collection methods and sources of data, including the review of program material and related literature, review of investor-owned utility (IOU) program data, telephone surveys, in-person interviews, in-depth interviews (in person or by phone), in-home audits, onsite visits, and stocking inventories. Most of the analyses relied on descriptive statistics, but
Changes to California's market effects evaluation protocol
All three studies recommended changes to California's market effects evaluation protocol, including allowing for the estimation of total net effects (includes free ridership, participant spillover, and nonparticipant spillover), and the use of Delphi panels as part of the basic level of rigor. More specific recommendations for each market are provided below.
Lessons learned
Many lessons were learned in the evaluation of market effects, and some of the most important were the following:
- •
Market effects need to be estimated throughout a program's life cycle.
- •
Baseline market data (pre-conditions as well as ongoing current market practice) need to be collected throughout a program's lifecycle—ideally, before program implementation.
- •
Because non-program (comparison areas) are becoming harder to find, timing is crucial and other methods will need to be used (e.g.,
Future market effects studies
In order to determine what further research should be done in the future in the area of market effects, CIEE conducted a survey in February 2011. Forty (40) individuals responded to the survey. Most of the respondents were consultants (38%) or from academia (28%), and many of them were professors (23%), evaluators or market researchers (20%), program planners or managers (18%), or evaluation or market research managers (15%). Since we do not know the size of the population of people who are
Conclusions
The three market effects studies were successful in collecting and analyzing a variety of data to meet the objectives of the market effects studies:
- 1.
Understand the cumulative effects of California's energy efficiency programs in three markets (CFLs, HBLs, and RNC)
- 2.
Quantify the savings caused by the above potential market effects for the 2006–2008 time period.
- 3.
Claim that the energy savings could be quantified with sufficient reliability to be claimed as a resource (HBL and RNC). However, while the
Recommendations
These studies affirm that measurement of program effects and market transformation is possible even in a market crowded with stimuli. But it is important to note that, as with other evaluation efforts, there is often a great deal of uncertainty when evaluating market effects. This uncertainty reflects the reliance of the evaluator on self-reports (of manufacturers, retailers, participating consumers, and nonparticipants) for assessing changes in the marketplace as well as program attribution.
Going forward—learning from lessons learned [add text]
Although the CPUC did not conduct a formal post project review, they did discuss the findings of these studies with their stakeholders and decided to conduct additional market effects studies. One study is on residential HVAC maintenance and installation, a second study is on LEDs, and a third study is on whole house construction. The studies will be prospective, rather than retrospective (although some retrospective elements will be part of the studies): They will develop comprehensive
Acknowledgements
This paper reflects the work of many people conducted over several years. First, I would like to thank CIEE's administrative staff for getting many of the contracts in place and for their administrative assistance over time (in particular, Jeanette Gorsira and Ken Krich). Second, I would like to thank the three advisors that helped CIEE in managing these contracts (Stephen Meyers, Ralph Prahl, and Isaac Turiel) and Ken Keating for providing evaluation oversight as an advisor to the CPUC. Third,
References (7)
California Energy Efficiency Evaluation Protocols: Technical, Methodological and Reporting Requirements for Evaluation Professionals
(2006)- KEMA and Itron, 2010. High Bay Lighting Market Effects Study. 〈http://uc-ciee.org/planning-evaluation/7/lbrsearch〉....
- KEMA (formerly RLW Analytics), NMR Group (formerly Nexus Market Research), Itron, and The Cadmus Group, 2010. Phase II...
Cited by (1)
The value of energy audits for SMEs: an Australian example
2016, Energy Efficiency
- 1
High bay lighting refers to a diverse group of technologies that are used to light spaces in commercial and industrial facilities with ceiling heights 15 ft and above.