[p1] High performance computing


Preliminary results for General I/O Arbitration Framework for applications using PnetCDF

T. V. Martsinkevich (RIKEN), W.-K. Liao (Northwestern University, USA), B. Gerofi (RIKEN), Y. Ishikawa (RIKEN), A. Choudhary (Northwestern University, USA), G.-Y. Lien (RIKEN), S. Nishizawa (RIKEN), T. Miyoshi (RIKEN), H. Tomita (RIKEN)


One way to tackle the growing volume of data that data analytics applications have to deal with is to apply HPC concepts to it including processing the data on large scale. One example of such application is SCALE-LETKF - a weather radar data assimilation application for real-time prediction of guerilla rainstorms - that is currently being developed at RIKEN. It is a two-component application in which the components exchange computation results via the Parallel File System. However, File I/O is known to impede scalability as increased volume of output data incurs longer execution time. In order to achieve timeliness in execution we developed a data arbitration framework that replaces File I/O with direct data transfering between the components. Using of the framework requires minimal modification of the application and the data transfering happens transparantly for the user. While the current implementation of the framework specifically targets applications that use PnetCDF for File I/O, we note that the framework is general and can be adapted to use any other I/O library.
We will present preliminary results that show significant speed-up of SCALE-LETKF using our framework compared to conventional use of File I/O.