CapTion
The CapTion toolkit is a set of tools for mixed immersive media productions that combine two forms of next-generation free-viewpoint video formats, namely, omnidirectional (360º) video and 3D media.
CapTion
The CapTion toolkit is a set of tools for mixed immersive media productions that combine two forms of next-generation free-viewpoint video formats, namely, omnidirectional (360°) video and 3D media. Starting from a multi-sensor capturing studio [1], presented in Figure 1, where spatially [2] and temporally aligned RGB-D videos are recorded. The performance of a user is captured and produced as a dynamic textured mesh [3], as showcased in Figure 2. The 3D content will be merged within the 360° media through the fusion post-production tool (Figure 3) that will exploit AI, and more specifically 360o scene understanding [4, 5], to allow for the seamless placement and realistic rendering of the 3D content within it [6].
References
[1] A low-cost, flexible and portable volumetric capturing system, V Sterzentsenko, A Karakottas, A Papachristou, N Zioulis, A Doumanoglou, D Zarpalas, P Daras, 2018 14th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS)
[2] Markerless Structure-based Multi-sensor Calibration for Free Viewpoint Video Capture, A Papachristou, N Zioulis, D Zarpalas, P Daras, Proceedings of 26th International Conference in Central Europe on Computer Graphics and Visualization (WSCG)
[3] Fast deformable model-based human performance capture and FVV using consumer-grade RGB-D sensors, DS Alexiadis, N Zioulis, D Zarpalas, P Daras, Pattern Recognition 79, 260-278
[4] OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas, N Zioulis, A Karakottas, D Zarpalas, P Daras, Proceedings of the European Conference on Computer Vision (ECCV), 448-465
[5] 360D: A dataset and baseline for dense depth estimation from 360o images, A Karakottas, N Zioulis, D Zarpalas, P Daras, 1st Workshop on 360o Perception and Interaction, European Conference on Computer Vision (ECCV)
[6] Hyper 360—Towards a Unified Tool Set Supporting Next Generation VR Film and TV Productions, B Takacs, Z Vincze, H Fassold, A Karakottas, N Zioulis, D Zarpalas, P Daras, Journal of Software Engineering and Applications 12, 127-148
Figure 1 The volumetric capture studio that allows for the multi-view calibration and synchronization of multiple RGB-D sensors.
Figure 2 The 3D performance capture production tool that uses multi-view RGB-D video recordings and produces a dynamic textured mesh by fitting a template model within the performance of the users.
Figure 3 The 360°-3D fusion tool that emplaces 3D performance capture content within pre-recorded omnidirectional scenes in equirectangular projection format. The female actor in red/black colors has been post-produced within the “crime scene” footage.