Big Data Assimilation

15-3 January 24 14:45-15:00

An Overview of DTF for Coupling Software Components

Y. Ishikawa (RIKEN), Tatiana V. Martsinkevich(RIKEN), Balazs Gerofi (RIKEN)


Applications integrating several software components use file IO to exchange data among those components especially in the case that such components are independently implemented. In sucha an application, the file IO processing time is usually not dominant for the total execution time.

However, if an application needs real-timeness, mitigating the file IO processing time must be considered. In order to mitigate this processing time, we have been developing a file IO library, called DTF, to support direct parallel data transfer between software components using the MPI communication libaray. DTF has been used in a high-resolution 30-minute numerical weather prediction system, using weather observable data that is available every 30 seconds.

This library dynamically creates simulation processes belonging to the simulation model, and assimilation processes belong to the assimilation model within a same MPI (Message Passing Interface) communicator, and then both kinds of processes can exchange their data via MPI communication rather than file-based I/O operations.

Contact information

Dr. Yutaka Ishikawa RIKEN