Immersive Capture System for Real-Time Photoreal Virtual Environment Rendering

The technology captures 360° video and audio using multiple cameras, processes the data into a 3D format, and blends real footage with computer graphics to create an interactive, photorealistic, navigable virtual environment.

Background

Currently, most standard video is shot from a single view by a single recording device. The view is predetermined by the recorder, and therefore a viewer of the video has no control over it. Recently there has been increasing popularity and utility of panoramic (360-degree) videos. In this format, a specially designed camera captures the entire scene surrounding it from a single location. A 360-degree video player plays back a smaller region of interest of the entire panorama at any one time, and allows the viewer to freely change which part of the panorama is displayed during playback. However, the viewer’s position is still limited to the single location in the scene from which the panoramic video was captured. There is no way to navigate to different vantage points of the scene.

Technology

Researchers at the University of Maryland have created a system to overcome the limitations of a single-view 360-degree video. The system utilizes multiple 360-degree video and audio setups that simultaneously record at different locations within a scene. This allows the viewer to navigate among a large number of captured or synthesized viewpoints. The start of the recording can be synchronized amongst the setups, or can be started in an unsynchronized manner and synchronized in a later pre-processing step. The recordings are then preprocessed, transcoded, and then used to create a fused multi-modal representation. Special rendering software decodes the multi-modal representation and presents the scene to the viewer. The software can also overlay a user interface to enable the viewer to interact with the scene. With the UI, the user can navigate to different viewpoints, annotate the scene, or select branching options.

Benefits

  • Enables immersive 360° video and audio experiences capturing multiple viewpoints.
  • Integrates real-world data with computer-generated graphics to maintain photorealism.
  • Supports dynamic interactions like real-time hologram reconstruction and 3D object insertion.
  • Provides a navigable and freely explorable virtual environment for users.

Applications

  • Virtual tour experiences
  • Real estate walkthroughs
  • Remote training simulation
  • Immersive gaming experiences

Additional info

A system comprising multiple cameras that record 360° video and audio is used to capture detailed real-world data. This data is preprocessed, compressed, and combined with computer-generated graphics to form a photorealistic 3D virtual environment. The technology supports real-time view-dependent hologram reconstruction and dynamic insertion of 3D objects, enabling interactive, freely explorable virtual settings on user devices.