ECE Assistant Professor Sheng Wei is the recipient of a new NSF award for the project "Content-Based Viewport Prediction Framework for Live Virtual Reality Streaming." This is a three-year $496,085 collaborative award led by Rutgers with Northeastern University and Texas State University. Rutgers' share of this award is $214,874.
In this project, Sheng and his team will develop a new content-based viewport prediction framework to improve the bandwidth and performance in live virtual reality (VR) streaming, which predicts the user's viewport through a fusion of tracking the moving objects in the video, extracting the video semantics, and modeling the user's viewport of interest. VR video streaming has been gaining popularity recently with the rapid adoption of mobile head mounted display (HMD) devices in the consumer video market. As the cost for the immersive experience drops, VR video streaming introduces new bandwidth and performance challenges, especially in live streaming, due to the delivery of 360-degree views. This project consists of three research thrusts. First, it develops a content-based viewport prediction framework for live VR streaming by tracking the motions and semantics of the objects. Second, it employs hardware and software techniques to facilitate real-time execution and scale the viewport prediction mechanism to a large number of users. Third, it develops evaluation frameworks to verify the functionality, performance, and scalability of the approach. The project uniquely considers the correlation between video content and user behavior, which leverages the deterministic nature of the former to conquer the randomness of the latter. With the rapidly increasing popularity of VR systems in domain-specific immersive environments, the project will benefit several VR-related fields of studies with significant bandwidth savings and performance improvements, such as VR-based live broadcast, healthcare, and scientific visualization. Moreover, the interdisciplinary nature of the project will enhance the education and recruitment of underrepresented minorities in several science, technology, engineering, and mathematics (STEM) fields.