Workforce development programs, such as Building Operator Certification, or BOC, represent a key strategy for achieving long-term behavior change in the energy sector. They also help ensure high-performance buildings operate as designed. But how do you measure success or attribute energy savings to training and certification?
Historically, research approaches have varied greatly. However, having conducted multiple impact assessments of BOC efforts in the last 20 years, we were able to develop a consistent methodology for claiming savings on training programs such as this.
While we tailor specific research objectives and tasks to client priorities, the overall approach includes interviews with program staff and trainers, as well as a review of program curricula, documentation, data, and logic models to identify intended behavioral outcomes. This research enables us to then develop comprehensive participant and, where appropriate, nonparticipant surveys to assess the operations and maintenance actions taken by program participants in response to what they learned through BOC. From there, we can estimate energy impacts resulting from their actions.
Historically, a key challenge of this research is that it is based on self-reporting of operations and maintenance, or O&M, practices by building operators. To address this, we carefully constructed a set of questions that identifies multiple levels of rigor in the performance of the various types of O&M activities for which building operators are responsible. This approach generates a precise but conservative estimate of energy savings for each surveyed operator.
What we learned:
We have applied this methodology in evaluations for the Northwest Energy Efficiency Alliance, the Northeast Energy Efficiency Partnerships, and Pacific Gas & Electric. Over the course of this work, several best practices surfaced. Namely:
- Depth and breadth enable greater accuracy. Evaluations that assess behaviors across a broad variety of activities and behaviors produce more accurate savings estimates because they provide an opportunity to assess all of the savings that occur.
- Consistency is key. Studies that assess savings by asking about practices in a consistent fashion across a wide range of equipment areas produce more savings than studies that are less rigorous and systematic.
- Overlap does not necessarily mean redundancy. In facilities with multiple building operators, the responsibilities of the multiple operators only overlap somewhat, with most of each operator’s responsibilities unique to that operator. So, multiple trained operators generate more savings than a single trained operator.
- Certification and training can lead to long-term savings. There is evidence that savings persist well beyond the expiration of certification. Our studies also point to the possibility that operators with advanced BOC training (Level 2) operators generate more savings than those with basic (Level 1) training.
Several program administrators throughout the country have adopted Research Into Action’s methodology for assessing the impacts of BOC training programs, including NEEA and NEEP. Insights from our research have also informed program design strategies to optimize program results and shape marketing strategies to successfully increase program participation and certification.
See our most recent Market Progress Evaluation for NEEA here.