Abstract
ORNL operates the world's brightest neutron source, the Spallation
Neutron Source (SNS). Funded by the US DOE Office of Basic Energy
Science, this national user facility hosts hundreds of scientists from
around the world, providing a platform to enable break-through
research in materials science, sustainable energy, and basic
science. While the SNS provides scientists with advanced experimental
instruments, the deluge of data generated
from these instruments represents both a big data
challenge and a big data opportunity. For example, instruments at the
SNS can now generate multiple millions of neutron events per second
providing unprecedented experiment fidelity but leaving the user with
a dataset that cannot be processed and analyzed in a timely fashion
using legacy techniques. To address this big data challenge, ORNL has
developed a near real-time streaming data reduction and analysis
infrastructure. The Accelerating Data Acquisition, Reduction, and
Analysis (ADARA) system provides a live streaming data infrastructure
based on a high-performance publish subscribe system, in situ data
reduction, visualization, and analysis tools, and integration with a
high-performance computing and data storage infrastructure. ADARA
allows users of the SNS instruments to analyze their experiment as it
is run and make changes to the experiment in real-time and visualize
the results of these changes. In this paper we describe ADARA, provide
a high-level architectural overview of the system, and present a
set of use-cases and real-world demonstrations of the technology.
