ADIOS

Exascale architectures will have complex, heterogeneous memory hierarchies, ranging from node-level caches and main memory all the way to persistent storage via the file system, that applications need to effectively achieve their science goals. At the same time, exascale applications are becoming more complex in their data flows, from multiscale and multiphysics simulations that need to exchange data between separate codes to simulations that invoke data analysis and visualization services to extract information and render it to storing simulation output to the file system for later analysis. The ADIOS project delivers a highly optimized coupling infrastructure that enables efficient synchronous and asynchronous data exchanges to move data between multiple codes running concurrently and to the different layers of the storage system. 

The Adaptable I/O Systems (ADIOS) is designed to tackle data management challenges posed by large-scale science applications running on high-performance computers that require, for example, code-to-code coupling for multiphysics and multiscale applications and code-to-service coupling for data analysis and visualization. Notably, ADIOS provides a simple, declarative publish/subscribe input/output interface so that applications can easily describe the data they produce or consume. ADIOS has multiple optimized solutions to transfer data between codes and to the file system. 

The ADIOS team is focused on providing high-quality, performant software products to address the data flow requirement of exascale applications on pre-exascale and exascale architectures. Therefore, the team is optimizing and creating new ADIOS microservices to efficiently perform the data transfers needed by applications that can use pre-exascale and exascale architecture features so that the exascale applications can meet their science and performance goals. The framework that the team delivers supports both synchronous and asynchronous data exchanges and provides the capabilities necessary for in situ processing and code coupling and efficient data transfers to the file system.

Progress to date 

• The ADIOS team released ADIOS2, which supports two file-based engines to write and read to/from permanent storage. The BPFile engine provides unmatched I/O performance on pre-exascale machines and supports in situ processing of data by multiple readers, while the HDF5 engine uses the parallel HDF5 library to write and read HDF5 formatted files without any overhead, matching the native library’s performance. The BPFile engine is used in several applications for highperformance storage I/O. In particular, the LAMMPS code in the EXAALT project is using ADIOS to dump its atoms at largescale runs, and ADIOS achieves better performance with a self-describing output that can be processed in situ than its original binary dump. 
• The team developed a flexible staging engine (SST) to allow for coupling codes via network communication that can be used for in situ processing where consumers of the data can dynamically connect to and disconnect from the producer’s (simulation) output. This engine has been used in code coupling production runs in the WDMApp project. 
• The team produced an MPI-based in situ engine for in situ data processing where data are moved from one application to another using asynchronous MPI send/ receive operations. This engine is used by applications to perform, for example, in situ data reduction using their ZFP and SZ lossy compression plugins. This engine is optimized for applications where the producer outputs data with a fixed schema at every step and the consumer reads the data with a fixed read pattern.