OmniCloud
The 360° video annotation process interacts with the OmniCloud component that physically hosts our database (Database management system), to store and retrieve information for a specific 360° video configuration as HotSpot details (e.g. for text: position, colour, shape, view options as “center of FOV”, “fixed on the screen”, and associated events).
The OmniCloud component also makes available to the OmniCap component a dedicated storage area, accessible via SFTP, where to transfer the 360° video files available for further annotation. When a curator decides to go public with a 360° enriched video file, making it available to user players, the OmniCloud component performs the following macro-actions:
- Copy the source 360° video file on a dedicated video streaming storage area (AWS Media Services).
- Set the 360° video configuration data as available for the players and no longer accessible for further edits/updates
Profiling
User profiling in Hyper360 is based primarily on the use of implicit feedback, in order to allow for seamless personalised content delivery
Automatic Camera Path
The goal of automatic camera path generation (automatic cinematography) is to calculate automatically a visually interesting camera path from a 360° video, in order to provide a traditional TV-like consumption experience
Reccomandation Engine
Hyper360’s Recommendation Engine drives the decisions as per the content that better matches each viewer’s preferences and provides related media recommendations
Semantic Interpreter
Content metadata are semantically interpreted into a set of fuzzy ontology-based concepts and axioms
Profiling Engine
Without requiring any manual effort on their side, the user’s state of immersion and engagement is further heightened through an experience that is more relevant to their preferences.
Pivotal to an effective implicit personalisation strategy is the use of 360° videos. While traditional, fixed perspective video offers little to no insight as to what viewers are looking at, free viewpoint video offers the unique opportunity to recognise areas of interest and disinterest. In addition, it allows all the added interactions of enhanced traditional video, such as interacting with superimposed objects and hyperlinks.
To this end, the viewing behaviour of the user, as captured through OmniPlayer, coupled with the information of what the interesting content is about (as characterised by content metadata authored through OmniConnect and stored in the video annotation DB) are the most pivotal aspects that ensure effectively personalised, highly immersive viewing experiences.
The basis for relating user choices in the UI to meaningful terms and a subsequent recommendation of suitable topics is the provision of meta tags to interactive UI elements (hotspots) and also to metadata fields. These fields cover areas of interest without interaction features. They are tagged in order to track which objects users look at. This allow to determine interest in specific features, even without an active click on a button.
Automatic Camera Path
An initial prototype of the automatic camera path generator was researched and developed, which is mainly based on the information about the scene objects (delivered from the object detection and tracking algorithm). For each scene object, a saliency score is calculated based on several influence factors (object class and size, motion magnitude, visited map, neighbours of object), which indicates the “interestingness” of the object. From the calculated saliency scores for the scene objects, an automatic camera path is generated for the current shot by tracking the currently most interesting object (the one with the highest saliency score).
Recommendation Engine
Hyper360’s Recommendation Engine drives the decisions as per the content that better matches each viewer’s preferences and provides related media recommendations. Recommendation is based on semantic matching between user profiles and content metadata. The application of recommendation is three-fold within Hyper360, offering personalised navigation within the 360o media, manifested both as cues for personalised camera path(s), as well as cues for personalised 3D Mentor narratives, while also involves targeted embedded hyperlink (advertisements) delivery
Semantic content interpreter
In order to have a holistic representation of content metadata and subsequently on the user preferences which are implicitly learned based on the metadata of the content that the user has consumed, and minimize loss of information, content metadata are semantically interpreted into a set of fuzzy ontology-based concepts and axioms. This interpretation is based on learned lexico-syntactic relations between metadata, with a method that supports domain-specific re-training based on the Hyper360 viewers’ community consumed metadata patterns.
