My summer at Sealed [guest post – Crystal Yeh]

Time seems to pass by in a blink of the eye, and before I knew it, the 10 weeks of my summer internship with Sealed has passed. I’ve learned so much about customer behavior, energy auditing, the relative frustrations of energy auditing software, as well as what makes a successful homeowner interaction during a neighborhood canvass or fair. 

Based on my experience, I think that people in the energy efficiency field need to ask more questions based on behavioral economics and marketing, not building science. In particular, we need to ask what motivates people to invest in energy efficiency improvements? There are many benefits to energy efficiency improvements: saving money, reducing emissions/being environmentally friendly, increasing comfort, increasing home value, and reducing noise pollution. (Most people don’t think of noise pollution as an efficiency benefit, but  when you think about it soundproof rooms are characterized by heavy insulation). 

Interestingly, I’ve found (via homeowner interviews, train interviews, canvassing interviews and fair interviews) that homeowners will rarely spontaneously speak about any of the non-monetary benefits except when asked about them specifically. Only when prompted about specific comfort issues, for example, will they reply “Yes, that too.” Everything besides savings is secondary in the mind’s of a homeowner. Money is king.

In a similar vein, how questions are phrased in interviews can make all the difference. Giving an answer prompt will yield different responses than having people respond to an open question. In order to have an unadulterated insight into the motivations of a homeowner, you have to listen to what the homeowner says without a prompt. When you bait a question (known in psychology as “priming”) people will be more likely to agree to ideas they would never have thought of on their own accord. Think about how much easier it was for you back in school to answer multiple-choice questions over free response questions. 

Priming, although not conducive to scientific surveys, is a great tool to use when marketing a product or service. To give an example, when I shadowed an audit yesterday, I noticed that the homeowner was presented with three different options for a new boiler, an 85% efficiency oil boiler, an 85% efficiency natural gas boiler, and a 95% efficiency natural gas boiler. With three different options, the homeowners has already mentally modeled having each of those options in their home and is committed to choosing at least one. 

The audit that I shadowed was one of the most education experiences I’ve ever had this summer. I watched the technician Hip perform the Carbon Monoxide test, the boiler efficiency test (this home was so old that the boiler was 60 years old), the width of wall insulation and the infrared gun test. The last test was performed with the homeowners watching and listening to the explanation, which I thought was great for improving the confidence and trust between contractor and customer. 

The audit helped crystallize one thing I learned this summer, which is that the traditional rational-choice economic model doesn’t necessarily hold up in practice. For example, the homeowners insisted that a proposed 35 gallon indirect water heater might not be sufficient for their family’s needs (“I don’t ever want to run out of hot water!” exclaimed the woman) even though it would be more than enough for them as it’s just a 3 person home. Based on this experience, I think more academics or policy makers should spend some time in the real world in order to better understand how real people think about energy and efficiency. 

These past few months have been so educational and interesting for me as a person. I really feel so empowered at the potential of energy efficiency, the low hanging fruit if you will, to lower Long Island, New York State, and eventually the nation’s emissions significantly. I also realized that I’m really going to miss the coworkers, partners and people I’ve met through this internship, especially my awesome boss extraordinaire, Andy Frank.

Why Homeowners Don’t Trust Energy Efficiency. And What Might Change Their Thinking.



Variations in home construction, age and other building factors influence the level of savings achieved. Frank points out that if the mean savings from a retrofit is $1,000 per year, any given home is as likely to save $400 as $1,600.

“It’s like putting your money in a tech stock,” Frank said.

Great new company taking the risk factor out of the equation so homeowners can actually weigh the cost-benefits of efficiency upgrades on balance.

Why Homeowners Don’t Trust Energy Efficiency. And What Might Change Their Thinking.

How existing audit software make things harder for Sealed

One thing that we’ve struggled with at Sealed is how to make it clear to homeowners how much money Sealed makes from the guarantee.

The truth is we take 20 to 25 percent of the “average actual” savings, which translates to $200 to $250 a year per home for a typical project that saves a home $1,000 a year.

However, the inaccuracy of existing audit software makes it difficult to explain this clearly. Because the audit tools generally over-predict savings, sometimes quite dramatically, our prospective customers often believe we are making much more money than we are.

In our pilot area of Long Island, for example, average projected savings are $1,300 a year, average actual savings are $1,000 and average guaranteed savings are $750 to $800. So based on the audit report, the homeowner thinks Sealed is making $500 a year or more (we wish!), when the truth is we are making less than half of that.

So why is most audit software over-predicting savings and what can be done about it?

Savings Over-Prediction

There are three main reasons why most energy audit software over-predict savings.

The first is simply that it’s very hard. As Mike Rogers relayed from #BScamp conference:

Some things that are hard to model: foundation heat loss, infiltration, wall and attic heat loss, window loss and gain, and HVAC performance. In other words, it’s hard to model homes, especially existing homes!

Michael Blasnik among others have demonstrated how inaccurate most building models are, and how greater model complexity simply leads to greater error.

The second type of error comes from entering and analyzing billing data. Most audit software leverages historical billing data, but can still overstate actual usage, both overall and within specific building systems.

The root of this problem is disaggregation, or breaking down the bill into its component parts, namely “base” and “weather-variable” load (e.g. heating & cooling). For example, if you are calculating the savings from a new high efficiency hot water system, you probably want to know how much energy is currently being used for hot water so that you can accurately model the impact of a more efficient system. 

Sounds simple, but when analyzing bills some audit software does non-intuitive things like assume the usage data entered is entirely for weather-variable load, and then making a separate calculation for base load (e.g. water heating). This causes the software to over-predict savings because it doesn’t correctly interpret the billing data, adding estimated base load to actual total usage. 

For example, for a standard oil to gas conversion, Conservation Services Group’s Real Home Analzyer software will often predict a home will save more oil than it uses. Try explaining that to a homeowner!

The third type of error is the “true-up” process to align billing data with the audit models, a blend of the first two problems. Because the audit models don’t often line up with actual data, auditors go back into the software to change the model until it roughly matches the real data. 

While this “true-up” process is better than nothing, the truth is that reality is often suspended (e.g. unrealistic indoor temperature settings) and sometimes the software will literally allow auditors to put in a fudge factor. PSD’s TREAT software, for example, has a “usage adjustment multiplier” for hot water demand that can be adjusted until the model is “trued-up”. 

What Can be Done?

So we have a situation where there is savings over-prediction from modeling, billing analysis and “true-up” error. What is to be done!

First, we have to distinguish between accuracy and precision. Today, audit software is neither accurate (correct on average) nor precise (consistent across homes). 

The first priority should be to make the software accurate so as not to systematically over-predict savings. Inaccurate savings estimates do a disservice to homeowners, contractors and ratepayers. And of course it also makes it harder for Sealed.

To improve the accuracy of audit software, a few things need to happen:

  1. Stop blaming “behavior” and the homeowner. There’s no evidence for dramatic behavioral change post-efficiency improvement.
  2. Stop creating incentives for contractors to treat the software like a video game to hit arbitrary cost-effectiveness benchmarks (in other words move from TRC to UCT
  3. Most importantly, measure the actual results and adjust the models accordingly. 

The first step is admitting this is a problem. Matt Golden, Michael Blasnik, and Mike Rogers, among other industry leaders, are confronting this issue head on, but we need the entire industry to try and improve audit software accuracy. As detailed in the last post, the key is #openenergydata, which requires a big push for regulators and other stakeholders to leverage existing data sets. 

The precision problem is where Sealed comes in. As we’ve shown, homes are difficult to model, period. So while it should be possible to get things right “on average”, average savings don’t help an individual homeowner. As Mike Rogers tweeted today in reference to LBL’s HES software

“Lots of scatter but both above and below the line” Sorry, that is no help in individual homes! 

Sealed solves this problem by guaranteeing the savings, smoothing out the risk across a large sample size of homes. 

But while we can be honest with homeowners, it makes our job more difficult when the Sealed savings guarantee is compared to over-stated savings projections. Turns out, it’s hard to be honest.