ECE Colloquium - October 18, 2013
Dr. Judy Qiu, School of Informatics and Computing, Indiana University at Bloomington
CoRE Lecture Hall
Many scientific applications are data intensive. It is estimated that organizations with high- end computing infrastructures and data centers are doubling the amount of data that they are archiving every year. Twister extends MapReduce, enabling HPC-Cloud Interoperability. We show how to apply Twister to support large-scale iterative computations that are common in many important data mining and machine learning applications. Furthermore, one needs additional communication patterns from those familiar in MapReduce. This leads us to the new Map-Collective programming model which we develop here based on requirements of a range of applications by extending our existing data analysis framework Twister. The resultant Map-Collective model, which captures the full range of traditional MapReduce and MPI features, is built on a new communication abstraction. It will be integrated with Hadoop and evaluated with Twister, HDinsight and Twister4Azure. Our work includes a detailed performance evaluation on IaaS or HPC environments such as Azure, FutureGrid and IU’s Big Red II supercomputer, and provides useful insights to both frameworks and applications.
Dr. Judy Qiu is an Assistant Professor of Computer Science in the School of Informatics and Computing at Indiana University and an assistant director of the school’s Digital Science Center. Her research interests are parallel and distributed systems, cloud computing, and high-performance computing. Qiu leads the SALSA project, involving professional staff and Ph.D. students from the School of Informatics and Computing. SALSA focuses on data-intensive computing at the intersection of cloud and multicore technologies with an emphasis on scientific data analysis applications by using MapReduce and traditional parallel computing approaches. Her research has been funded by NSF, NIH, Microsoft, Google and Indiana University. She is a recipient of NSF CAREER Award in 2012