Visual improvement of decoded point clouds rendering in extended reality

13 Avril 2023

---

Catégorie : Doctorant

---

Context and problematic

The immersive technologies (virtual reality VR, augmented AR and mixed MR, these terms are grouped here under the name of extended reality XR) require the development of new data formats for the acquisition and representation of natural 3D objects. Point clouds are promising for rendering static and dynamic 3D objects. However, point clouds require a very large amount of data and it is therefore necessary to compress them to facilitate processing, sharing and/or display. However, the coding/decoding operations introduce distortions altering the point clouds and therefore their rendering. The integration of these 3D contents into complex virtual (VR) or real (AR/MR) scenes also requires to think about the influence of ambient lighting on the perception of the rendering of decoded point clouds in immersive scenes.

Many studies focus on the visual perception of the final rendering of these 3D objects, or as a corollary, research aims to optimize elements of the overall processing chain of these contents, from their capture to their final rendering. However, to our knowledge, the perceived quality of point clouds is always evaluated in non-interactive and non-immersive scenes, on videos where the rendering of the cloud is done in poor, unlit scenes, and on a predefined visual trajectory for the observer.

For this PhD work, we therefore propose to study the final rendering of 3D point clouds in immersive and interactive scenes, in particular we will be interested in the perceived quality of these contents according to the light characteristics of the scene and to their restitution in order to optimize the processing carried out (representation, compression / decompression, re-lighting, rendering, etc.).

 

Objectives

The objective of the thesis is to contribute to the improvement of the visual rendering of 3D point clouds decoded in XR. A thorough state of the art about objective and subjective approaches for evaluating the visual quality of point clouds rendered in this context is to be done, in particular if we consider a free observation in immersion within the 3D scene. The selection of a suitable corpus of contents for the experimental tests will be necessary. The research will initially focus on the study of the influence of lighting on the masking (or conversely on the facilitation) of the perception of decoding artefacts. An interesting point will be then to optimize the final rendering of immersive content by considering more particularly the most efficient coding/decoding schemes and representation formats for point clouds, and a computation of the lighting of the rendered scene adapted to best hide artifacts and optimize the observer's quality of experience according to different identified use cases.