How existing audit software make things harder for Sealed

One thing that we’ve struggled with at Sealed is how to make it clear to homeowners how much money Sealed makes from the guarantee.

The truth is we take 20 to 25 percent of the “average actual” savings, which translates to $200 to $250 a year per home for a typical project that saves a home $1,000 a year.

However, the inaccuracy of existing audit software makes it difficult to explain this clearly. Because the audit tools generally over-predict savings, sometimes quite dramatically, our prospective customers often believe we are making much more money than we are.

In our pilot area of Long Island, for example, average projected savings are $1,300 a year, average actual savings are $1,000 and average guaranteed savings are $750 to $800. So based on the audit report, the homeowner thinks Sealed is making $500 a year or more (we wish!), when the truth is we are making less than half of that.

So why is most audit software over-predicting savings and what can be done about it?

Savings Over-Prediction

There are three main reasons why most energy audit software over-predict savings.

The first is simply that it’s very hard. As Mike Rogers relayed from #BScamp conference:

Some things that are hard to model: foundation heat loss, infiltration, wall and attic heat loss, window loss and gain, and HVAC performance. In other words, it’s hard to model homes, especially existing homes!

Michael Blasnik among others have demonstrated how inaccurate most building models are, and how greater model complexity simply leads to greater error.

The second type of error comes from entering and analyzing billing data. Most audit software leverages historical billing data, but can still overstate actual usage, both overall and within specific building systems.

The root of this problem is disaggregation, or breaking down the bill into its component parts, namely “base” and “weather-variable” load (e.g. heating & cooling). For example, if you are calculating the savings from a new high efficiency hot water system, you probably want to know how much energy is currently being used for hot water so that you can accurately model the impact of a more efficient system. 

Sounds simple, but when analyzing bills some audit software does non-intuitive things like assume the usage data entered is entirely for weather-variable load, and then making a separate calculation for base load (e.g. water heating). This causes the software to over-predict savings because it doesn’t correctly interpret the billing data, adding estimated base load to actual total usage. 

For example, for a standard oil to gas conversion, Conservation Services Group’s Real Home Analzyer software will often predict a home will save more oil than it uses. Try explaining that to a homeowner!

The third type of error is the “true-up” process to align billing data with the audit models, a blend of the first two problems. Because the audit models don’t often line up with actual data, auditors go back into the software to change the model until it roughly matches the real data. 

While this “true-up” process is better than nothing, the truth is that reality is often suspended (e.g. unrealistic indoor temperature settings) and sometimes the software will literally allow auditors to put in a fudge factor. PSD’s TREAT software, for example, has a “usage adjustment multiplier” for hot water demand that can be adjusted until the model is “trued-up”. 

What Can be Done?

So we have a situation where there is savings over-prediction from modeling, billing analysis and “true-up” error. What is to be done!

First, we have to distinguish between accuracy and precision. Today, audit software is neither accurate (correct on average) nor precise (consistent across homes). 

The first priority should be to make the software accurate so as not to systematically over-predict savings. Inaccurate savings estimates do a disservice to homeowners, contractors and ratepayers. And of course it also makes it harder for Sealed.

To improve the accuracy of audit software, a few things need to happen:

  1. Stop blaming “behavior” and the homeowner. There’s no evidence for dramatic behavioral change post-efficiency improvement.
  2. Stop creating incentives for contractors to treat the software like a video game to hit arbitrary cost-effectiveness benchmarks (in other words move from TRC to UCT
  3. Most importantly, measure the actual results and adjust the models accordingly. 

The first step is admitting this is a problem. Matt Golden, Michael Blasnik, and Mike Rogers, among other industry leaders, are confronting this issue head on, but we need the entire industry to try and improve audit software accuracy. As detailed in the last post, the key is #openenergydata, which requires a big push for regulators and other stakeholders to leverage existing data sets. 

The precision problem is where Sealed comes in. As we’ve shown, homes are difficult to model, period. So while it should be possible to get things right “on average”, average savings don’t help an individual homeowner. As Mike Rogers tweeted today in reference to LBL’s HES software

“Lots of scatter but both above and below the line” Sorry, that is no help in individual homes! 

Sealed solves this problem by guaranteeing the savings, smoothing out the risk across a large sample size of homes. 

But while we can be honest with homeowners, it makes our job more difficult when the Sealed savings guarantee is compared to over-stated savings projections. Turns out, it’s hard to be honest.

Leave a Reply

Your email address will not be published. Required fields are marked *