Runtime System for I/O staging in support of Voluminous in-situ Processing of extreme scale data

g-extreme-338x101.jpg

Prof. Manish Parashar received a DOD grant on the project "RSVP: Runtime System for I/O staging in support of Voluminous in-situ Processing of extreme scale data".   Prof. Manish Parashar received this grant in a collaborative profject with Oak Ridge National Laboratory, Lawrence Berkeley National Laboratory, Rutgers University and Georgia Tech.   The RU portion is $300K.

RSVP: Runtime System for I/O staging in support of Voluminous in-situ Processing of extreme scale data

Abstract

Advances toward the exascale era are forcing re-examination of scientific data management. Interfaces used between computation, I/O, and storage must provide unprecedented levels of flexibility in how I/O actions are carried out: to enable online data analytics, to deal with increasingly complex memory hierarchies, and to support new programming paradigms. Online analytics demand service-oriented models of computing in which analytics pipelines are easily composed, reused, and efficiently run alongside the scientific simulations producing extreme-scale data. Increasingly, there are also demands imposed by experimental data used to validate simulation results at scale. In response to these needs, the RSVP project will develop a model in which computational and analytic services can be easily and efficiently associated with and applied to science data as part of an end-to-end, in-situ "process flow".

Leveraging the flexibility and composability of the service-based approach already ubiquitously employed in modern web processing systems, abstract formulations of process flows will make it easy for scientists to formulate the analytics and I/O actions needed to gain desired scientific insights. Flexibility in creating "concrete process flows" from abstract process flows will offer opportunities to match diverse exascale hardware and to efficiently exploit complex memory hierarchies. To aid this process, a rich set of ``data transformations'' helps organize, reorganize, filter, index, and compress data to match simulation vs. analytics needs and appropriately use storage technologies. Complementing these features, automated runtime management will deal with process flow dynamics and make it possible to concurrently run the multiple process flows required by complex science codes such as domain-specific Uncertainty Quantification (UQ) operating at exascale.

Building on the early successes of the Adaptable I/O System (ADIOS) framework, in both I/O performance and using staging for in-situ and in-transit data analytics, the design and realization of process flows will be done in coordination with the many science teams and will include specific application drivers---materials science, seismology, and fusion---chosen to each advance certain process flow technologies. The proposed effort will also be informed by a continuing and extensive, multi-year engagement with both U.S. and international science teams. The desired outcome of the coordinated effort is a service-oriented platform that permits scientists to retain control over the flow of data and data analytics, while allowing them to abstract away from the extraneous details of their actual data management plans.