Abstract
Variability in the performance of shared system components is a major obstacle in analyzing the effective throughput of leadership class computers. Shared file systems and networks are serious impediments to achieving repeatable application performance on HPC systems. In particular, performance analysts are likely to be interested in quantifying differences between average-case behavior, worst-case behavior, and standard deviation for shared system components. Typical descriptions of these statistics assume a normal distribution; however, in non-linear and multi-modal performance distributions, summary statistics are often misleading. In this paper we describe Confidence, a tool for analyzing the full spectrum of performance for a benchmarking code. By including all of the experimental outcomes in the analysis without discarding any measurements, Confidence enables a novel analysis of benchmark performance. Finally, we use Confidence to characterize the network performance of three high performance computer systems, and we provide recommendations for leveraging network performance in latency sensitive applications.