Achievement: A multidisciplinary team of researchers from Duke University, New Jersey Institute of Technology, Oak Ridge National Laboratory (ORNL), Lawrence Livermore National Laboratory, and University of Texas at El Paso has developed an adaptive physics refinement (APR) technique to effectively model cancer cell transport. The APR integrates multiphysics modeling by limiting the region in which cellular-scale dynamics are realized to the volume surrounding a cell of interest—termed the “window”, which has the benefits of capturing fluid-structure interaction that could not be explained purely through hemodynamics, reducing the amount of computationally expensive high-resolution mesh, and limiting the locations in which fluid- structure interaction is needed. In this work, researchers present multiple validations of the APR framework by comparing against fully resolved fluid-structure interaction methods and employ techniques, such as latency hiding and maximizing memory bandwidth, to effectively utilize heterogeneous node architectures available in the current DOE supercomputers such Summit at OLCF and Lassen at LLNL.
Significance and Impact: The ability to track simulated cancer cells through the circulatory system, important for developing a mechanistic understanding of metastatic spread, pushes the limits of today’s supercomputers by requiring the simulation of large fluid volumes at cellular-scale resolution. To overcome this challenge, we introduce a new adaptive physics refinement (APR) method that captures cellular-scale interaction across large domains and leverages a hybrid CPU-GPU approach to maximize performance. Through algorithmic advances that integrate multi-physics and multi-resolution models, we establish a finely resolved window with explicitly modeled cells coupled to a coarsely resolved bulk fluid domain. Collectively, the computational developments and performance optimizations performed by this work provide a robust and scalable framework to enable system-level simulations of cancer cell transport.
Research Details
We address the challenges in building a coupled multiphysics capability to be broadly used for circulatory modeling by 1) introducing a multi-block scheme to couple the coarse and fine lattices, 2) creating a framework for splitting scales between a tightly coupled CPU-based bulk model and a GPU-based window model, 3) developing algorithms to move the window, and 4) efficiently distributing work between the host and device on a heterogeneous accelerator-based machine such as the Summit supercomputer while maximizing overlap of communication and computation. As demonstrated in the evaluation results, the APR method proposed in this work radically increases the potential domain sizes for multiphysics modeling to enable new classes of problems to be tackled with in silico methods.
Facility: Simulations for performance analysis were performed on the Summit supercomputer at ORNL and the Lassen supercomputer at LLNL.
Sponsor/Funding: DOE ASCR, NSF, NIH
PI and affiliation: Amanda Randles, Department of Biomedical Engineering, Duke University.
Team: Daniel F. Puleri (Duke University), Sayan Roychowdhury (Duke University), Peter Balogh (New Jersey Institute of Technology), John Gounley (ORNL), Erik W. Draeger (LLNL), Jeffrey Ames (Duke University), Adebayo Adebiyi (Duke University), Simbarashe Chidyagwai (Duke University), Benjamın Hernandez (ORNL), Seyong Lee (ORNL), Shirley Moore (University of Texas at El Paso), Jeffrey S. Vetter (ORNL)
Citation and DOI: Daniel F. Puleri, Sayan Roychowdhury, Peter Balogh, John Gounley, Erik W. Draeger, Jeffrey Ames, Adebayo Adebiyi, Simbarashe Chidyagwai, Benjamın Hernandez, Seyong Lee, Shirley Moore, Jeffrey S. Vetter, Amanda Randles, High Performance Adaptive Physics Refinement to Enable Large-Scale Tracking of Cancer Cell Trajectory, IEEE Cluster Conference (Cluster), 2022.
Summary: The ability to track simulated cancer cells through the circulatory system, important for developing a mechanistic understanding of metastatic spread, pushes the limits of today’s supercomputers by requiring the simulation of large fluid volumes at cellular-scale resolution. To overcome this challenge, we introduce a new adaptive physics refinement (APR) method that captures cellular-scale interaction across large domains and leverages a hybrid CPU-GPU approach to maximize performance. Through algorithmic advances that integrate multi-physics and multi-resolution models, we establish a finely resolved window with explicitly modeled cells coupled to a coarsely resolved bulk fluid domain. In this work we present multiple validations of the APR framework by comparing against fully resolved fluid-structure interaction methods and employ techniques, such as latency hiding and maximizing memory bandwidth, to effectively utilize heterogeneous node architectures. Collectively, these computational developments and performance optimizations provide a robust and scalable framework to enable system-level simulations of cancer cell transport.
