Should you tune up your behavioral energy program?

Jan 25, 2016
How often do you change the oil in your car? Why not more frequently or less frequently? Most of us don’t give this question a lot of thought – we just follow the recommendations of the vehicle manufacturer or our auto mechanic (or at least try to). The recommendations we follow are actually based on a benefit-cost analysis that seeks to strike an appropriate balance between vehicle performance and service costs over time. You could certainly change your car’s oil every 500 miles and it would provide excellent protection against engine wear and tear. However, the amount of time and money spent on oil changes would outweigh these benefits. A reasonable interval is one where the engine protection and associated performance and durability benefits outweigh the material and labor costs of the oil change.
 

DSM program administrators face a similar question when implementing behavioral conservation programs. Industry studies have shown that the frequency and duration of exposure to behavioral messaging are positively correlated with savings estimates. But this additional exposure increases the cost to deliver the program. At what point are behavioral programs changing the oil every 500 miles? What is the optimal program delivery model? The answer depends heavily on assumptions about the persistence of behavioral savings and the savings accounting mechanism employed by the jurisdiction. In this post I examine the implications of persistence on program design and delivery given different regulatory or accounting constraints and I introduce potential implementation strategies to help optimize program performance.

Home Energy Reports (HERs) are a form of behavioral conservation that has become increasingly utilized by DSM program administrators in recent years to achieve energy savings in the residential sector. HERs achieve savings by raising awareness of energy consumption and inducing cognitive dissonance among customers who are using more energy than similar households. This stimulus is coupled with energy-saving recommendations to give motivated recipients discrete actions that they can take to reduce energy consumption. Energy savings from HER programs have been studied extensively and generally produce statistically significant reductions of 1-2%. Now that the treatment effect is reasonably well understood, industry attention has increasingly shifted to examining the persistence of HER savings. Assumptions about persistence – or how long HER energy savings will continue to materialize at the meter – are a critical input in cost-effectiveness calculations.

A 2014 White Paper by the Cadmus Group synthesized findings from HER implementations across the US where mailings had been purposely discontinued in order to analyze the effect on energy savings. The paper went on to recommend a departure from traditional methods for calculating lifetime savings and the associated avoided costs. Equipment measures are generally assumed to produce the same level of energy savings each year until the equipment reaches the end of its expected mechanical life, or measure life. Cadmus proposed a “decay” perspective for HER programs where the treatment effect gradually dissipates over time, resulting in reduced savings each year until the effect erodes completely. They hypothesized that an average annual decay rate of 20% was reasonable given the empirical evidence. Figure 1 contrasts the two approaches over time for two offerings implemented in 2015 that achieve identical lifetime savings.

Figure 1: Lifetime Savings Trajectory of HERs vs. Equipment Measures

Nexant analyzed several additional persistence experiments in 2015 and found results that were generally consistent with previous analyses. In one analysis, treatment group homes were saving an average of 1.7% of total electric use 16 months after receiving the last HER. Findings like this have spurred utilities, vendors, and regulators alike to question the one-year measure life that had traditionally been utilized to calculate cost-effectiveness and progress towards savings goals. With a decay-based perspective on the horizon, Nexant began to examine the effect of revised assumptions on savings potential and portfolio planning and how client utilities could modify HER program delivery to optimize effectiveness under these revised constraints.

Figure 2 illustrates the challenge that traditional HER program delivery models will face using a decay-based perspective. Because the majority of savings from the first-year of HER exposure are assumed to persist in subsequent years, the incremental savings achieved by mailing home energy reports to the same customers is limited to the avoided decay and any growth in the treatment effect. Under this perspective, program benefits comfortably outweigh the costs for the first year of the program, but in years two through five the program does not pass the TRC test. The values shown in Figure 2 are for illustration only as each jurisdiction has different avoided cost values, per-home impacts, discount rate assumptions, and even preferred benefit cost tests. However, the key point is that a decay based perspective calls into question whether or not it is a prudent use of program funds to issue HERs to the same homes year after year. In the hypothetical example shown in Figure 2, the TRC ratio for the 5-year initiative is 1.0 – meaning that no net benefit was achieved by operating the program.

Figure 2: Continued Exposure with Decay

The Vaccination Model

Nexant believes a shift in HER program delivery is needed to optimize cost-effectiveness given what we know about the persistence of the treatment effect. We have nicknamed one potential approach the “vaccination” model because it mimics the viewpoint taken by public researchers when recommending the frequency of inoculation for diseases. In this delivery model, homes would receive HERs for some period of time and then have the treatment discontinued. In this example, all homes receive mailings in 2015 and have then reports stop in 2016. This allows the program to capture the initial, highly cost-effective period of exposure and then foregoes the subsequent years when the cost-effectiveness of issuing HERs is marginal. Once savings have decayed to a point where it is once again cost-effective to treat customers, HERs are issued again for some period of time. In this example, the second wave of treatment occurs in 2019 and savings that persist through 2023 are monetized in the benefit-cost ratio. The TRC ratio for the 5-year initiative is 1.43.

Figure 3: Vaccination Model

 

The Crop-Rotation Model

One drawback of the vaccination model is that it produces inconsistent savings from year to year. In the example shown in Figure 3, the program only achieves incremental annual (first-year) savings in 2015 and 2019. Figure 4 shows another potential HER delivery model that produces more consistent savings year-on-year. In the “crop rotation” model, eligible customers are identified and then randomly assigned to a treatment and control group. Then the groups are further subdivided into cohorts based on the assumed decay rate. In this example an average annual decay rate of 20% is assumed and the eligible accounts are divided into five distinct cohorts. The premise of the crop rotation model is to treat a group of customers for one year and then withdraw the treatment until savings have completely decayed. Only one of five treatment cohorts will actually receive HERs during any given year. The TRC ratio for this 5-year initiative is 2.29. The crop rotation maximizes TRC by ensuring all savings are first-year savings; however, it does so at the expense of aggregate first-year and lifetime energy savings.

 Figure 4: Crop Rotation Model

Adjusting these models for cumulative annual accounting constraints

The vaccination and crop rotation models are viable options in jurisdictions which rely on incremental annual accounting to assess goal achievement. Incremental annual accounting essentially sums the first-year savings achieved during each year of a program cycle, so the year in which the savings occur is irrelevant when assessing progress towards goals. Other jurisdictions rely on a cumulative annual accounting method that places artificial importance on the year in which savings occur and discourages implementation of short-lived measures early in the program cycle. Under cumulative annual accounting, only incremental annual savings that have not reach the end of their EUL count towards goals. With this accounting perspective, assumptions about persistence and decay become less important and the accounting method is the key constraint. In our 5-year program cycle example above, incremental annual savings of 100 kWh from an HER program implemented in 2015 would contribute nothing towards goals, but 100 kWh from a 2019 implementation would contribute 100 kWh towards goals. Program administrators in jurisdictions with cumulative annual accounting must take this account and craft an appropriate delivery model. Assuming the goal is to maximize energy savings towards goals per program dollar, program administrators should postpone behavioral program implementation until the year prior to the goal assessment date.