[10-2] February 28, 17:10-17:30

A Flexible I/O Arbitration Framework for netCDF based Big Data Processing Workflows on High-End Supercomputers

J. Liao (Southwest University of China), B. Gerofi (RIKEN), G. Lien (RIKEN), S. Nishizawa (RIKEN), T. Miyoshi (RIKEN), H. Tomita (RIKEN), and Y. Ishikawa (RIKEN)

We present an initial study of a direct communication framework designed for complex workflows that eliminates unnecessary file I/O among components. Specifically, we propose an I/O arbitrator layer that provides direct parallel data transfer among job components that rely on the netCDF interface for performing I/O operations, with only minimal modifications to application code. We present the design and an early evaluation of the framework on the K Computer using up to 4800 nodes running RIKEN's experimental weather forecasting workflow as a case study.