Photo-Realistic Human Modeling for Real-World Human Cognition by Care-Robots
5 Octobre 2021
Catégorie : Stagiaire
Robot vision for human cognition often fails to work well in the real-world situation, despite the disruptive results achieved in Computer Vision and Artificial Intelligence. While most training data have been collected in well-conditioned, easy-to-isolate backgrounds, wild videos from the real-world may contain various environmental conditions such as lighting, background patterns, and, most notoriously, occlusions. The latter becomes the source of recurrent problems of human cognition by care-robots in the in-house situation. Large variations in body shapes, motions, clothes, and frequent interactions with objects also contribute to the difficulty. Finally, viewing dynamics by moving robots and humans is another source of the problem. Learning-based methods relying on dataset inevitably show limited performance, as it is almost impossible to collect a large, annotated dataset that spans over all possible configurations of the real-world scene.
Model-based learning approaches are good alternatives to this problem, as have been confirmed by a considerable amount of previous works on human face and body recognition from images based on pre-constructed 3D models. However, many existing models consider only the geometry of human face or body in isolation, making it difficult to real-world situation involving obstacles or environmental objects. In this project, we will also adopt a model-based approach but with enhanced realism, additional dimension (i.e. time), and beyond. Our aim is to push the current limits of robot vision in human cognition by care-robots in the in-house situation. The specific goal is to make the performance of the vision-intelligence robust to large variations (in body shapes, motions,..) to occlusion (cloth, furniture, wall,..), and capable of understanding the interaction by developing a photo-realistic, physics-aware 4D human model.
This main objective of this internship is to develop a photorealistic human model, by enhancing the realism of an existing 3D human model. We will focus on the two following aspects:
1.Adding color- and illumination-components to an existing model. Among others, SMPLgeometric human model is considered: the body model is parameterized by the pose vector and shape vector , with a template mesh whose pose-dependent deformation is computed using a linear blend skinning function. Although it is common to assume that the color- and illumination components are independent from the geometry, we will consider the correlation between the geometry and the color in this work. All models will be developed in a differentiable manner, so that an inverse problem can be formulated and solved later.
2.Automatic dressing the human model with ready-made clothes. We will develop an algorithm for automatic resizing and dressing of multiple cloth models, coming from public or in-house datasets. This may require a soft integration with our cloth reconstruction module and/or a physically based cloth simulator, depending on the design choice.
All developments will be made in an open-source simulation environment for robot learning.
* Candidate profile
— Master student in Computer Science or in (Applied) Mathematics
— Solid programming skills: Python/C++
— Background in geometric modeling and statistics
— Good communication skills
 Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34, 6, Article 248 (November 2015), 16 pages.