Blind Triangulation: Seeing the Entire “Spillover“ Elephant

Spillover has been on my mind a lot lately. And, if you’re in a jurisdiction where net savings matter, then it should be on yours as well. Based on a recent literature review, I’ve come to the following conclusions: Everybody in the evaluation community thinks spillover should be included in net-to-gross. However, it frequently is omitted, and when it is included, the most common approaches almost guarantee it is underestimated.

While there is variation in how spillover is assessed, very often assessments resemble one (or both) of the following: We ask program participants or nonparticipants how much the efficiency programs influenced their behavior outside the program. Or (and) we ask vendors and contractors how much efficient equipment they sold to nonparticipants and how much the program influenced those sales.

I’m not criticizing anyone who’s done the above. I’ve done it. I’ve recommended others do it. But, I’ve come to realize that each of these approaches ask for something that respondents can only partially provide.

Customers can speak to a program’s direct influence (how program marketing, outreach, or participation taught them the value of energy efficiency). Conversely, vendors and contractors may speak to the program’s indirect influence (how the program influenced their recommendations and their sense of how much their recommendations, in turn, influenced their customers). But, just as customers can’t see the program’s indirect influence via the vendors and contractors, the latter can’t see the program’s direct influence on their customers.

It’s like the story of the blind men and the elephant; we get multiple perspectives, none of which alone represents the big picture.

My colleagues at ADM Associates and I recently developed an approach to reveal that big picture, which I presented at IEPEC. It starts with the recognition of multiple scenarios of un-incented efficient equipment sales, each representing a separate “pot” of potential spillover. Each one allows for the possibility of direct influence (some allow only direct influence) and some allow for indirect influence via a vendor, contractor, or both:

So, say a contractor buys recommended equipment from a vendor, and then recommends and sells that equipment to a customer. There are three possible influence pathways:

We define the indirect influence in each pathway as the product of the influence (measured from 0% to 100%) of each actor (program, vendor, contractor) on the next. In scenarios with multiple influence pathways, the one with the greatest influence determines the spillover savings in that pot. And why is that?

Suppose I recommended a new restaurant to you. Because you don’t know my tastes, you ignore the recommendation, but you later try it on a mutual friend’s recommendation. However, suppose I’m the one who recommended the restaurant to our mutual friend. While I had little direct influence on your dining choice, I had significant indirect influence – and, therefore, I had significant influence. You just didn’t know it.

So, how did our new approach work out? From a survey of vendors and contractors, we identified 17 GWh of un-incented efficient lighting sales for a nonresidential program across the five scenarios. After calculating the program influence in each scenario, we found 12 GWh of that to be spillover, equal to 12% of the gross reported lighting savings.

I’m not saying we found the whole elephant, but we did a lot better than if we’d just measured the tail.