An Integrated Middleware Framework to Enable Extreme Collaborative Science

Dr. Shantenu Jha received funding from the Department of Energy's Office of Advanced Scientific Computing Research (ASCR) for his project "An Integrated Middleware Framework to Enable Extreme Collaborative Science". Dr. Jha is the PI and there are two collaborators, Drs. D. S. Katz from the Univ. of Chicago and J. Weissman from the Univ. of Minnesota.

This project aims to bridge the gap between application requirements and diverse and heterogeneous platforms, by developing a middleware framework that can support the needs of tools and services in support of distributed scientific collaborative applications at extreme scales. We consider a variety of distributed

applications, including those that operate on rich data pipelines in a distributed collaborative environment including: data generation and capture, data preprocessing, data analysis, and data storage and delivery. For these applications, tools and services are needed at all levels of this pipeline to enable data discovery, data transmission and streaming, data placement and storage, resource discovery, computation scheduling, and co-scheduling. Other types of applications share most of these needs. A framework-based middleware provides an integrated way to address the many co-dependent issues in extreme-scale environments such as the emergence of disparate resource platforms and network capabilities, the inherent distribution of compute and data, and multiple-levels of application and run-time decision making.

We propose a middleware framework that provides powerful abstractions for distributed computational and storage resources, and containers for computational tasks and distributed data. This project will address fundamental research challenges required to realize these abstractions, including techniques to enable resilience and performance for both data and computation. Our framework will enable a varied set of tools to be more easily constructed.