Les commentaires sont clos.

CIFRE Huawei/UGE/IGN: Multi-view image processing in degraded outdoor conditions

5 December 2022

Catégorie : Doctorant

3 years founded PhD program with Huawei, University Gustave Eiffel and IGN on NeRF in degraded outdoor conditions for autonomous driving applications.

PhD Thesis Proposal:

Multi-view image processing in degraded outdoor conditions

This PhD is a CIFRE fellowship between Huawei, the CoSys department of University Gustave Eiffel and the french
Institut national de l'information géographique et forestière (IGN).
Huawei is working on key components of L2-L3 autonomous driving platform and is progressively shifting focus to
the development of breakthrough technologies required for L4-L5 levels. Tomorrow self-driving cars powered by AI
will combine edge and cloud computing with vast number of sensors to safely drive customers and deliver
merchandise. At Huawei, we develop realistic simulators created from crowd-sourced data to continuously improve
localization, perception and prediction algorithms of autonomous vehicles. We are seeking the best candidates for
a CIFRE PhD with a background in computer vision, deep learning, simulation, computer graphics, mapping,
perception, sensor fusion, cognition and other related areas, to work as a part of IoV team in Paris Research Center
(PRC). As a member IoV PRC you will closely work with multiple teams worldwide to grow your expertise and
successfully transfer your research results into real products.
Created in 2020, the University Gustave Eiffel brings together a research institute, a university, a school of
architecture and three engineering schools focusing on the study of urban areas. The Cosys department
(Components & Systems) focuses on urban mobility, from the design to the evaluation of innovative systems likely
to improve urban experience. In this Department, the PICS-L lab has a strong experience on vision issues, including
ADAS and autonomous vehicles applications.
The French mapping agency IGN (National Institute for Geographic and Forest Information) is a public administrative
establishment attached to the French Ministry of Ecological Transition; it is the national reference operator for
mapping the French territory. The LaSTIG* Laboratory in Sciences and Technologies of Geographic Information for
the smart city and sustainable territories, is a joint research unit attached to the Gustave Eiffel University, the IGN
and the Paris Engineering School (EIVP). It is a unique research structure in France and even in Europe, bringing
together around 80 researchers, who cover the entire life cycle of geographic or spatial data, from its acquisition to
its visualization, including its modeling, integration and analysis; among them about thirty researchers work in image
analysis, computer vision, machine learning, photogrammetry and remote sensing.

Research topic:
The core subject of this PhD focuses on evaluating and improving state-of-the-art image processing algorithm based
on the Neural Radiance Field (NeRF) implicit representation [1] for different tasks such as surface reconstruction,
simultaneous localization and mapping, novel view synthesis, in degraded outdoor conditions such as bad weather,
low illumination and/or dirty sensors.
Image processing in an outdoor environment is a challenging task due to many artifacts. Moving objects (vehicles,
pedestrians) or exposure changes are usually well tackled [2]. On the other hand, current algorithms are not always
robust to bad weather conditions and very low illumination. This can be explained by multiple causes:
• Bad weather conditions are less common than the perturbations aforementioned;
• The lack of datasets and benchmarks of scenes featuring degraded conditions for the evaluation and
training of learned-based methods;
• Artifacts and disruptors caused by poor outdoor conditions may be very challenging.
For example, fog or rain generates contrast reduction and specularities that require a specific attention. In [3], the
veil caused by a foggy weather is explicitly handled by adding a physical model of the fog visual effect. The modeling
of the visual effect of rainy weather has been also studied [4].
With the NeRF approach new avenues of research appear to handle fog and rain thanks to the modeling of
transparent object through the whole reconstructed volume. Impressive results in low light conditions are already
obtained [5] but only for static scene. The case of dirty sensors is rarely studied. In parallel, recent frameworks
simulate better weather effects and deliberately degraded visual data for driving scenario [6], enabling both learning
and evaluation of new methods on various conditions.
In this PhD thesis, we thus propose to evaluate the ability to handle bad weather conditions, low illumination and
dirty sensors with implicit representation NeRFs. This evaluation will rely on synthetic and real datasets as in [7]
where a fog machine was used. The resulting algorithms will also be tested under clean weather conditions, to check
for consistency.

1 Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2021). NeRF: Representing
scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 99-106.
2 Martin-Brualla, R., Radwan, N., Sajjadi, M. S., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021). Nerf in the
wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (pp. 7210-7219).
3 Caraffa, L., & Tarel, J. P. (2012). Stereo reconstruction and contrast restoration in daytime fog. In Asian conference
on computer vision (pp. 13-25). Springer, Berlin, Heidelberg.
4 Bossu, J., Hautière, N., & Tarel, J. P. (2011). Rain or snow detection in image sequences through use of a histogram
of orientation of streaks. International journal of computer vision, 93(3), 348-367.
5 Mildenhall, B., Hedman, P., Martin-Brualla, R., Srinivasan, P. P., & Barron, J. T. (2022). Nerf in the dark: High
dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (pp. 16190-16199).
6 Sun, T., Segu, M., Postels, J., Wang, Y., Van Gool, L., Schiele, B., ... & Yu, F. (2022). SHIFT: A Synthetic Driving
Dataset for Continuous Multi-Task Domain Adaptation. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (pp. 21371-21382).
7 Duminil, A., Tarel, J.-P. & Brémond, R. (2022). A new Real-World Video Dataset for the comparison of Defogging
Algorithms. In Proc. ASPAI' 2022: 4th International Conference on Advances in Signal Processing and Artificial
Intelligence. Kerkyra, Greece, October 19-21, 2022.

Description of research activities:
• Study the state of the art on 3D reconstruction in degraded condition and 3D reconstruction with implicit
neural representation.
• Identify bottleneck in 3D reconstruction in degraded visual conditions
• Propose new solutions for solving problems bring by degraded visual conditions based on implicit representations.
• Research and develop algorithm based on the proposed solutions
• Apply the proposed algorithm to the domain of self-driving cars using existing or specifically collected datasets
• Publish research results in top journals and conferences and participate to scientific seminars

This PhD will be supervised jointly between Huawei Technologies France, the LaSTIG laboratory of IGN (Paris area)
and the Cosys department (PICS-L lab) of the Université Gustave Eiffel.

The candidate should be motivated to carry out world class research and should have a Master in Computer Science,
with a focus on Vision and/or Robotics. He/She should have solid skills in the following domains:
• Implement Code in Python, C++ (CUDA is a plus)
• Apply or use existing libraries for deep learning in project related tasks (pytorch is a plus)
• Good knowledge in Computer Vision, Computer Graphics, 3D reconstruction and robotics
• Good knowledge in Git, ROS, OpenCV, Boost, multi-threading, CMake, Make and Linux systems
• Code and algorithm documentation
• Project reporting and planning
• Writing of scientific publications and participation in conferences
• Fluency in spoken and written English; French and/or Chinese is a plus
• Intercultural and coordination skills, hands-on and can-do attitude
• Interpersonal skills, team spirit and independent working style

Nathan Piasco (Huawei) –
Roland Brémond (thesis advisor, Univ. GE) –
Laurent Caraffa (IGN) –

Application deadline:

Application Files:
CV + motivation letter + transcript of records for academic years 2021-2022 and 2022-2023 + any other relevant