Annonce

CIFRE PhD Proposal: Multi-view image processing in degraded outdoor conditions with NeRF

5 Avril 2023

Catégorie : Doctorant

PhD Thesis Proposal:

Multi-view image processing in degraded outdoor conditions with NeRF

Background:

This PhD is a CIFRE fellowship between Huawei, the CoSys department of University Gustave Eiffel and the french Institut national de l'information géographique et forestière (IGN).

Huawei is working on key components of L2-L3 autonomous driving platform and is progressively shifting focus to the development of breakthrough technologies required for L4-L5 levels. Tomorrow self-driving cars powered by AI will combine edge and cloud computing with vast number of sensors to safely drive customers and deliver merchandise. At Huawei, we develop realistic simulators created from crowd-sourced data to continuously improve localization, perception and prediction algorithms of autonomous vehicles. We are seeking the best candidates for a CIFRE PhD with a background in computer vision, deep learning, simulation, computer graphics, mapping, perception, sensor fusion, cognition and other related areas, to work as a part of IoV team in Paris Research Center (PRC). As a member IoV PRC you will closely work with multiple teams worldwide to grow your expertise and successfully transfer your research results into real products.

Created in 2020, the University Gustave Eiffel brings together a research institute, a university, a school of architecture and three engineering schools focusing on the study of urban areas. The Cosys department (Components & Systems) focuses on urban mobility, from the design to the evaluation of innovative systems likely to improve urban experience. In this Department, the PICS-L lab has a strong experience on vision issues, including ADAS and autonomous vehicles applications.

The French mapping agency IGN (National Institute for Geographic and Forest Information) is a public administrative establishment attached to the French Ministry of Ecological Transition; it is the national reference operator for mapping the French territory. The LaSTIG* Laboratory in Sciences and Technologies of Geographic Information for the smart city and sustainable territories, is a joint research unit attached to the Gustave Eiffel University, the IGN and the Paris Engineering School (EIVP). It is a unique research structure in France and even in Europe, bringing together around 80 researchers, who cover the entire life cycle of geographic or spatial data, from its acquisition to its visualization, including its modeling, integration and analysis; among them about thirty researchers work in image analysis, computer vision, machine learning, photogrammetry and remote sensing.

Research topic:

The core subject of this PhD focuses on evaluating and improving state-of-the-art image processing algorithm based on the Neural Radiance Field (NeRF) implicit representation [1] for different tasks such as surface reconstruction, simultaneous localization and mapping, novel view synthesis, in degraded outdoor conditions such as bad weather, low illumination and/or dirty sensors.

Image processing in an outdoor environment is a challenging task due to many artifacts. Moving objects (vehicles, pedestrians) or exposure changes are usually well tackled [2]. On the other hand, current algorithms are not always robust to bad weather conditions and very low illumination. This can be explained by multiple causes:

·Bad weather conditions are less common than the perturbations aforementioned;

·The lack of datasets and benchmarks of scenes featuring degraded conditions for the evaluation and training of learned-based methods;

·Artifacts and disruptors caused by poor outdoor conditions may be very challenging.

For example, fog or rain generates contrast reduction and specularities that require a specific attention. In [3], the veil caused by a foggy weather is explicitly handled by adding a physical model of the fog visual effect. The modeling of the visual effect of rainy weather has been also studied [4]. More recently, NeRF in combination with classical computer graphics methods has also been used for realistic adversarial outdoor condition data generation [5,6].

With the NeRF approach new avenues of research appear to handle fog and rain thanks to the modeling of transparent object through the whole reconstructed volume. Impressive results in low light conditions are already obtained [7], but only for static scene. While NeRF-powered approach has been recently introduced to tackle the problem of multi-sensor spatiotemporal calibration [8], dealing with data from degraded or dirty sensors is rarely studied. In parallel, recent frameworks simulate better weather effects and deliberately degraded visual data for driving scenario [9], enabling both learning and evaluation of new methods on various conditions.

In this PhD thesis, we thus propose to evaluate the ability to handle bad weather conditions, low illumination and dirty sensors with implicit representation NeRFs. This evaluation will rely on synthetic and real datasets as in [10] where a fog machine was used. The resulting algorithms will also be tested under clean weather conditions, to ensure consistency of the proposed method.

References:

[1] Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2021). NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 99-106.

[2] Martin-Brualla, R., Radwan, N., Sajjadi, M. S., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021). Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7210-7219).

[3] Caraffa, L., & Tarel, J. P. (2012). Stereo reconstruction and contrast restoration in daytime fog. In Asian conference on computer vision (pp. 13-25). Springer, Berlin, Heidelberg.

[4] Bossu, J., Hautière, N., & Tarel, J. P. (2011). Rain or snow detection in image sequences through use of a histogram of orientation of streaks. International journal of computer vision, 93(3), 348-367.

[5] Li, Y., Lin, Z. H., Forsyth, D., Huang, J. B., & Wang, S. (2022). ClimateNeRF: Physically-based Neural Rendering for Extreme Climate Synthesis. arXiv preprint.

[6] Haque, A., Tancik, M., Efros, A. A., Holynski, A., & Kanazawa, A. (2023). Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions. arXiv preprint.

[7] Mildenhall, B., Hedman, P., Martin-Brualla, R., Srinivasan, P. P., & Barron, J. T. (2022). Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16190-16199).

[8] Herau, Q., Piasco, N., Bennehar, M., Roldão, L., Tsishkou, D., Migniot, C., ... & Demonceaux, C. (2023). MOISST: Multi-modal Optimization of Implicit Scene for SpatioTemporal calibration. arXiv preprint.

[9] Sun, T., Segu, M., Postels, J., Wang, Y., Van Gool, L., Schiele, B., ... & Yu, F. (2022). SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21371-21382).

[10] Duminil, A., Tarel, J.-P. & Brémond, R. (2022). A new Real-World Video Dataset for the comparison of Defogging Algorithms. In Proc. ASPAI' 2022: 4th International Conference on Advances in Signal Processing and Artificial Intelligence. Kerkyra, Greece, October 19-21, 2022.

Description of research activities:

·Study the state of the art on 3D reconstruction in degraded condition and 3D reconstruction with implicit neural representation.

·Identify bottleneck in 3D reconstruction in degraded visual conditions

·Propose new solutions for solving problems bring by degraded visual conditions based on implicit representations.

·Research and develop algorithm based on the proposed solutions

·Apply the proposed algorithm to the domain of self-driving cars using existing or specifically collected datasets

·Publish research results in top journals and conferences and participate to scientific seminars

This PhD will be supervised jointly between Huawei Technologies France, the LaSTIG laboratory of IGN (Paris area) and the Cosys department (PICS-L lab) of the Université Gustave Eiffel.

The candidate should be motivated to carry out world class research and should have a Master in Computer Science, with a focus on Vision and/or Robotics. He/She should have solid skills in the following domains:

·Implement Code in Python, C++ (CUDA is a plus)

·Apply or use existing libraries for deep learning in project related tasks (pytorch is a plus)

·Good knowledge in Computer Vision, Computer Graphics, 3D reconstruction and robotics

·Good knowledge in Git, ROS, OpenCV, Boost, multi-threading, CMake, Make and Linux systems

·Code and algorithm documentation

·Project reporting and planning

·Writing of scientific publications and participation in conferences

·Fluency in spoken and written English; French and/or Chinese is a plus

·Intercultural and coordination skills, hands-on and can-do attitude

·Interpersonal skills, team spirit and independent working style

Nathan Piasco (Huawei) – nathan.piasco@huawei.com

Roland Brémond (thesis advisor, Univ. GE) – roland.bremond@univ-eiffel.fr

Laurent Caraffa (IGN) – laurent.caraffa@ign.fr

Application deadline:

Application Files: