Development of criteria to evaluate autonomous vehicle agents is an ongoing problem. Led by Robert Patton, Group Lead of the Learning Systems Group, a team at ORNL has designed new evaluation metrics to better assess the quality of a given agent. The team uses CARLA, an open-source driving simulator designed for autonomous vehicle development. Existing metrics implemented for CARLA typically measure completions on a provided route, excluding details of the quality of driving. To differentiate between various driving behaviors, the team developed a unique scoring system. The first version of the metric focused on four driving characteristics: distance driven, ability to stay in the center of the lane, how much the vehicle weaves, and speeding infractions. The penalties and rewards were calculated using positional information provided by CARLA, and each factor was given a different weight. The team tested the metric by comparing calculated scores of a set of agents to the ratings given by a group of human evaluators. Over several iterations, the metric was refined to the point where it consistently agreed with the human evaluators. The team is now looking to expand the complexity of behaviors that the metric evaluates, such as traffic light infractions. [Shang Gao et al., “Quantitative Evaluation of Autonomous Driving in CARLA,” IEEE Intelligent Vehicles Symposium, 2021. DOI: TBD]
