SHARE
Research Highlight

Level Set Learning for Reducing Uncertainties in Function Approximation

Topics:
Level Set Learning for Reducing Uncertainties in Function Approximation CSMD Computer Science and Mathematics Division ORNL
Figure 1: Comparison of neural network approximation of 𝑓=π‘₯_1^2+〖… +π‘₯γ€—_20^2 𝑖𝑛 Ξ©=[0,1]^20 with and without using the method. (Left column): the method completely avoided the overfitting issue, and the relative error was 1.6%. (Middle column): direct neural network regression results in severe overfitting and a 35.1% relative error. (Right column): direct neural network regression with more hidden neurons leads to much worse overfitting and a 51.3% relative error.

The Science
ORNL researchers developed a novel nonlinear level set learning method to reduce dimensionality in high-dimensional function approximation. This method: 

  • Develops reversible neural networks (RevNets) and pseudo-reversible neural networks (PRNNs) to handle level sets with different types of geometries.
  • Introduces new loss functions to train nonlinear transformations, i.e., RevNets or PRNNs, to learn low-dimensional parameterization of the level sets. 
     
Level Set Learning for Reducing Uncertainties in Function Approximation CSMD Computer Science and Mathematics Division ORNL
Figure 2: Level set learning and function approximation results produced by the method for 𝑓=π‘₯_1^2 βˆ’π‘₯_2^2 in Ξ©=[βˆ’1,1]^2, where the critical point is at [0, 0]. Although the RevNet model cannot learn the level sets of this function using a reversible transform, the PRNN model successfully learned the level sets as shown in the quiver plot (left). The gradient field (blue arrows) are perpendicular to the red vector field learned by the PRNN model.

The Impact

  • This method significantly alleviates the overfitting issue when approximating a high-dimensional function with limited data (see Figure 1).
  • The method successfully learns level sets of functions with critical points, such as saddle points, in the input domain (see Figure 2).

Team Members: Guannan Zhang, Jacob Hinkle, Jiaxin Zhang (all of ORNL), Yuankai Teng, Zhu Wang, Lili Ju (all of the University of South Carolina), Anthony Gruber (Florida State University).

Publication(s) for this work:

  • G. Zhang, J. Hinkle, J. Zhang, β€œLearning nonlinear level sets for dimensionality reduction in function approximation.” Advances in Neural Information Processing Systems, 32, 2019. 
  • Y. Teng, et al., β€œLeve set learning with pseudo-reversible neural networks for nonlinear dimension reduction in function approximation.” Submitted to SIAM Journal on Scientific Computing (https://arxiv.org/pdf/2112.01438.pdf) 
     

Summary
One of major uncertainty resources in building predictive models is the limited information (i.e., training data) when the input space is high-dimensional. In this case, an effective way to reduce uncertainty is to reduce the dimension of the input space. The question the team tried to answer is how to effectively reduce the dimension of a function with independent input variables. The researchers developed two types of nonlinear level set learning models during the project. In FY20, they developed a RevNet-based level set learning method, where the key novelty is the definition of the new loss function for training the RevNets. The RevNet-based method provided high performance (see Figure 1), but the team found that it cannot handle functions with critical points, such as the saddle points shown in Figure 2. In FY21, the researchers improved their method by replacing the RevNet with a PRNN. Since PRNN is not reversible, they added a new regularization term in the loss to encourage the pseudo reversibility. With the use of the PRNN, this method can learn level sets that have critical points, which has not been achieved by state-of-the art dimension reduction methods.

Acknowledgement of Support
This material was based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Applied Mathematics program under contract and award numbers ERKJ352 at Oak Ridge National Laboratory (ORNL). ORNL is operated by UT- Battelle, LLC., for DOE under Contract DE-AC05-00OR22725.