SHARE
Research Highlight

A Prediction Interval Method for Machine Learning Model Uncertainty Quantification

Topic:

Brief: Researchers have developed a distribution-free, computationally efficient, and practically reliable prediction interval method to quantify machine learning (ML) model prediction uncertainty.

Accomplishment:  Researchers have developed a prediction interval method (PI3NN) to quantify ML model prediction uncertainty and theoretically proven that it precisely captures the uncertainty for a given confidence level and completely avoids the crossing issues suffered by the state-of-the-art methods for a regression problem 𝑩 = 𝑓(đ‘„) + 𝜀.  PI3NN is particularly suitable for scientific ML (SciML). The produced uncertainty bound can assess the model prediction’s credibility and trustworthiness; it can identify data/domain shift and understand when the learned model could fail and why it fails; and it also can guide data collection to automate the experimental design and inform decision making. We developed PI3NN to accurately quantify model prediction uncertainty by training three neural networks (NNs), one NN to approximate the prediction value, the other two NNs to estimate the lower and upper bound of the prediction interval. The method directly communicates uncertainty using intervals which provides understandable information for decision making.  PI3NN does not introduce any unusual hyperparameters resulting in stable performance and it requires no distributional assumption on data which makes it widely applicable for credible predictions in natural sciences and safe operation and control in complex engineered systems.  

Top panels illustrate the four steps of our PI3NN algorithm. Bottom panels illustrate the effectiveness of the Out-of-Distribution (OOD) identification feature. CCSD AI Initiative
Top panels illustrate the four steps of our PI3NN algorithm. Bottom panels illustrate the effectiveness of the Out-of-Distribution (OOD) identification feature. Step1 trains one NN to obtain the prediction mean value; Step 2 adds a shift v to approximate median and generate samples for training the other two NNs; Step 3 trains the other two NNs to learn the upper and lower bound of the interval; and Step 4 precisely determines the prediction interval via root-finding method. When turning on the OOD identification feature, PI3NN can accurately identify the OOD regions [-7, -4] âˆȘ [4, 7] by giving them increasingly large prediction intervals as their distance from the training data gets large. If we turn off the OOD identification by using the default initialization, PI3NN will not identify the OOD regions by giving them a narrow uncertainty bound.

Acknowledgement: This research was funded by the AI Initiative, as part of the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC, for the U.S. Department of Energy (DOE).    

Publications resulting from this work: 

  1. Zhang, P., Liu, S., Lu, D., and Zhang G., A prediction interval method for uncertainty quantification of regression models. Published at ICLR 2021 workshop on simulation and deep learning. https://simdl.github.io/papers.
  2. Zhang, P., Liu, S., Lu, D., Sankaran, R., and Zhang, G., An out-of-distribution-aware autoencoder model for reduced chemical kinetics. Published in American Institute of Mathematical Sciences Journal. Doi: 10.3934/dcdss.2021138.
  3. Liu, S., Zhang, P., Lu, D., and Zhang G., PI3NN: Out-of-distribution-aware prediction intervals from three neural networks. Submitted to ICLR 2021. https://arxiv.org/abs/2108.02327.

Presentations resulting from this work:

  1. [Invited talk] Lu, D., “Uncertainty Quantification for Machine Learning Prediction” Invited speaker at the Department of Mathematics, Federal University of Parana (UFPR), Brazil, June 25, 2021, Virtual.
  2. [Invited talk] Lu, D., “A prediction interval method for uncertainty quantification of regression models” Invited speaker at the Society for Industrial and Applied Mathematics (SIAM) Annual meeting, July 22, 2021, Virtual.
  3. [Invited talk] Lu, D., “Machine Learning Methods for Improving Terrestrial Ecosystem Predictions” Invited speaker at the Ecological Society of America (ESA) Annual Meeting, August 2021, Virtual.

Contact: Dan Lu (lud1@ornl.gov)

Team: Dan Lu, Siyan Liu, Pei Zhang, Guannan Zhang