The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop:HPCSYSPROS20
Authors: Honwai Leong (DataDirect Networks (DDN)), Andrew Janke (University of Sydney), Daniel Richards (DataDirect Networks (DDN)), and Stephen Kolmann (University of Sydney)
Abstract: This paper presents the architecture of a highly parallelized data replication workflow implemented at The University of Sydney that forms the disaster recovery strategy for two 8-petabyte research data storage systems at the University. The solution leverages DDN’s GRIDScaler appliances, the information lifecycle management feature of the IBM Spectrum Scale File System, rsync, GNU Parallel and the MPI dsync tool from mpiFileUtils. It achieves high performance asynchronous data replication between two storage systems at sites 40km apart. In this paper, the methodology, performance benchmarks, technical challenges encountered and fine-tuning improvements in the implementation are presented.
Back to HPCSYSPROS20 Archive Listing