Topics:
![This graphic illustrates the compression ratio regression of the two compressors in different samplings. Computer Science and Mathematics Division CSMD ORNL](/sites/default/files/styles/large/public/2021-05/estimating_lossy_compressibility_of_scientific_data_using_deep_neural_networks.png?itok=WWaDQZvz)
This graphic illustrates the compression ratio regression of the two compressors in different samplings.
The Science
A team of ORNL researchers built a deep neural network to estimate the compressibility of scientific data and show that adding compressor-specific features can greatly improve the performance of prediction. To achieve this, the researchers:
- extracted features from the data characteristics or the inner mechanisms of compressors;
- compared the performance of compressor-specific features under different samplings; and
- compared the performance of deep learning with the sampling and analytical methods.
The Impact
The novel neural network:
- represents a keystone component that will enable the best uses of standard compressors on scientific data and allow researchers to better manage massive data sets.
- performs better than the previous standard of predicting data compressibility (using biased estimation and a white-box analytical model).
PI(s)/Facility Lead(s): Scott Klasky (ORNL)
Publication: Qin, Zhenlu, et al. Estimating Lossy Compressibility of Scientific Data Using Deep Neural Networks. IEEE Letters of the Computer Society 3.1 (2020): 5-8. DOI: https://doi.org/10.1109/LOCS.2020.2971940.Facility: Work was performed at Oak Ridge National Laboratory
Funding: DOE ASCR
![CCSD Communications Manager Scott Jones](/sites/default/files/styles/staff_profile_image_style/public/2023-06/badge%20photo.jpg?h=fdc857c4&itok=Df0f0kRg)