Identifying Outcome Measures to Evaluate Drivers’ Knowledge of Vehicle Automation

Abstract

Introduction

Advanced vehicle technologies have the potential to improve safety and convenience for drivers; however, to realize these benefits, it is important that drivers use the systems appropriately. Past research has documented gaps in drivers’ understanding of how new technology works and when it should be used. It follows that the interplay between a driver’s knowledge, performance, and safety outcomes, and approaches for training and education have become topics of interest in the research and stakeholder communities. An area needing further exploration is the relationship between a driver’s knowledge of the system and a wide variety of performance and safety outcome measures.

This study focused on drivers’ knowledge of adaptive cruise control (ACC), a technology that is currently deployed and widely available. The study was comprised of two parts: (1) a review, enumeration, and appraisal of measures of performance, safety and behavior that potentially bear on knowledge of advanced vehicle technology, and (2) an experimental study to explore and validate outcome measures that can be implemented in research or other settings to measures drivers’ knowledge and understanding of advanced vehicle systems.

Methodology

In Part 1, a gathering and cataloging of existing behavioral, performance, and safety measures was carried out via a review of the relevant scientific literature, engineering standards, and available datasets. Additionally, a workshop was convened with subject matter experts to discuss a variety of topics related to education, training, and measurement. Through an iterative process, the universe of measures gathered was refined and consolidated into a smaller set.

In Part 2, a driving simulator study was carried out at two sites with a common approach, scenarios, and measures in order to validate outcome measures that can be implemented to measure driver knowledge and understanding of advanced vehicle systems. Sixty-five participants completed the study across the two sites, where they used an ACC system in a variety of driving conditions, including takeover scenarios, non-takeover scenarios, system interactions, and routine driving. Their knowledge of ACC was assessed prior to the experimental drives. A machine learning approach was used to explore the relationship between outcome variables and the participants’ knowledge of ACC.

Key Findings

Part 1:

• A matrix of outcome measures was produced that documents different properties of the measures along with an appraisal of the advantages and disadvantages relative to their implementation in the context of drivers’ knowledge of vehicle technology.
• The identified measures varied greatly in terms of their implementation and complexity, and each category or measure has advantages and disadvantages with respect to its relevance to system knowledge and to training and education.
• The use of combinations or clusters of outcome measures would offer a stronger and more stable insight into a driver’s underlying knowledge, accounting for some of the inherent shortcomings of just a single measure or class of measures.

Part 2:

• Measurement windows or epochs that involved system interactions, control takeovers, or routine situations where the system is operating showed stronger outcomes in the modeling exercise (i.e., were more useful in predicting driver knowledge of ACC).
• Measures of eye glance behavior featured prominently in all of the relevant models (takeover scenarios, non-takeover scenarios, system interactions, and routine driving). The advantages of eye glance metrics could be due to the mapping onto a driver’s roles and responsibilities while using the technology.
• Vehicle control and safety related measures were not as prominent, except in cases where active vehicle control was necessary.

Recommendations:

• Employing a cluster of measures is advised to account for some of the limitations associated with individual measures.
• In cases where there are constraints in what types of information can be gathered or weighed, researchers and other stakeholders should do the following:
  • Prioritize measures of eye glance behavior to corroborate driver knowledge of the system; vehicle control and safety measures are more effectively applied in specific edge case or takeover situations.
  • Establish measurement windows around system interactions, system takeovers or disengagements (control transitions), and normal system operation where the system is activated.
  • Leverage subjective measures such as confidence or technology acceptance whenever possible, which are strong predictors of knowledge.
  • Incorporate or consider information about the driver, such as driving experience and demographics.