Stage de fin d'études ou Master 2 - Visual Tracking using Drones and Event cameras

24 Octobre 2023

---

Catégorie : Stagiaire

---

Context

This project will be conducted in the context of LEASARD project [1] which aims to increase the navigation autonomy of drones as Unmanned Aerial Vehicles (AUV) in search and rescue scenarios. Towards this goal, LEASARD project is dedicated to the enhancement of sensing and processing capabilities through the integration of event cameras and the utilization of appropriate deep neural networks to process data from these cameras.

Description and objectives

Our primary focus in this project is to perform visual tracking using drones equipped with event cameras. While object tracking in standard videos has been extensively studied [2], the challenges are notably different in drone (aerial) videos. This is due to the fact that target objects are often smaller, and their appearances can change significantly due to variations in scale, orientation, and viewing angles [3].

To address these challenges, we plan to leverage newly emerging bio-inspired sensors known as Dynamic Vision Sensors (DVS) or event cameras [4]. In contrast to conventional RGB cameras, DVS sensors capture motion information with minimal latency (in the order of microseconds), are almost free from motion blur and require less energy and computational resources. Additionally, DVS sensors outperform conventional cameras in terms of dynamic range (140 dB versus 60 dB), allowing them to operate effectively even in adverse lighting or weather conditions. Thanks to these unique properties, there has been a growing interest in harnessing these new sensors for autonomous drone navigation.

In the initial stage, the candidate will be tasked with evaluating existing visual tracking methods for processing data from event cameras. If necessary, the candidate will also explore the potential integration of conventional RGB cameras into the tracking process. In this evaluation, he candidate will have access to a recent large-scale DVS tracking benchmark [5]. The primary goal of this stage is to identify the most efficient solution for visual tracking using event cameras. In the subsequent stage, the chosen solution will be adapted to perform effectively in drone-captured images, despite the previously mentioned challenges. To assess the performance of the proposed object tracking system, experiments using the aerial dataset VisDrone [6] will be conducted with appropriate data processing to simulate event-based inputs. The exploration of event simulators, such as AirSim or ESIM, could be potentially investigated as part of this assessment. The control of drones for target following will be addressed in a separate project.

Candidate profile

• The candidate will be pursuing his/her last year of Master's or engineer’s degree. The balance between research and development will be determined based on the candidate's profile.
• A strong level of Python programming is required.
• An interest in deep learning frameworks (Pytorch) is also required.
• Good oral and written communication skills in English.

How to apply

Interested candidates are encouraged to send their applications (detailed CV, transcripts, and diplomas) as soon as possible to the following address:
hajer.fradi@imt-atlantique.fr

References

[1] LEASARD project, https://project.inria.fr/leasard/

[2] Marvasti-Zadeh, S. M., Cheng, L., Ghanei-Yakhdan, H., & Kasaei, S. (2021). Deep learning for visual tracking: A comprehensive survey. IEEE Transactions on Intelligent Transportation Systems, 23(5), 3943-3968.

[3] Hamdi, A., Salim, F., & Kim, D. Y. (2020, July). Drotrack: High-speed drone-based object tracking under uncertainty. In 2020 IEEE international conference on fuzzy systems (FUZZ-IEEE) (pp. 1-8). IEEE.

[4] Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., ... & Scaramuzza, D. (2020). Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1), 154-180.

[5] Wang, X., Li, J., Zhu, L., Zhang, Z., Chen, Z., Li, X., ... & Wu, F. (2021). Visevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybernetics 2021.

[6] Zhu, P., Wen, L., Du, D., Bian, X., Fan, H., Hu, Q., & Ling, H. (2021). Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 7380-7399.