2y post-doc Multi-Modal Deep Learning for the Analysis of 3D Forestry Data , LaSTIG IGN Paris
29 Septembre 2020
Catégorie : Post-doctorant
Keywords : 3D point clouds, multispectral imagery, deep learning, segmentation, clas- sification, forest strata, morphologicaldescriptors.
Workplace : LaSTIG lab. (IGN/UGE/EIVP) - Team STRUDEL 73avenuedeParis94165SaintMand´e,FRANCE
- IGN), Loic Landrieu (LaSTIG -IGN).
Motivation and background
This project aims to automatically derive forest biodiversity metrics from remote sensing multi-modal data acquisitions (images and 3D point clouds). These metrics are related to the complexity of vegetation structure and the spatial patterns of species. They are of utter impor- tancefor1)bettermodelingtheforestmicro-climate,takingintoaccountthespatialpatternsof vegetation structure, and 2) studying the relationship between observed vegetation biodiversity and derived biodiversity metrics describing the 3D structure of theforests.
The characterization of the 3D structure of vegetation and, in particular, the different layers of vertical stratification (trees, shrubs, forest floor) is essential for modeling the relationship betweenmicro-climaticandbiodiversityvariables(Huangetal.,2014;DaviesandAsner,2014). Currently, this characterization remains very coarse and imprecise : main approaches do not capture the full complexity of forestecosystems.
Theverticalstratificationofforestcanopiesistraditionallyconductedusingthedensitypro- fileanalysismethod(HolmgrenandPersson,2004),modellingvegetationprofilebyprobability distributionfunctions,eitherinaunimodalfashion(Deanetal.,2009)orwithamixtureoffunc- tions (Jaskierniak et al., 2011). Other solutions are based on standard segmentation algorithms of3DLiDARpointstodiscriminatemainvegetationlayers(Morsdorfetal.,2010;Ferrazetal., 2016).
State-of-the-art methods are parametric and lack genericity. They only rely on local vertical point distribution and neglect the relationship between vegetation layer, individual trees, their species and their characteristics. Since many of these morphological and biological descriptors havelatentsemantics,whichisnotwell-modeledbycurrentmethods,weproposetoaddressthe problem with a machine-learning, neural network-basedapproach.
The post-doctoral work will focus on the automated analysis of 3D vegetation from a com- bination of 3D point clouds and multispectral multi-date geospatial imagery. More precisely, airborne LiDAR data with a high density of points (60 pts/m2) will be used jointly to multis-pectral Sentinel-2 time series and very high resolution UAV images.
The tasks expected of the finalized network are as follows :
—Semantic segmentation of the 3D point clouds into tree species ;
—Instance segmentation of individual trees and of horizontal strata;
—Derivation of morphological indicator of individual trees, such as crown diameter, stem diameter, maximal height, thickness, andextent.
Theproblemisamulti-modalandmulti-taskproblem,whichcombineshighdensity point clouds, superspectral time-series, and high definition images. The interleaved goals are to segment and classify the trees, and produce morphological descriptors. The different tasks of this post-doctoralcanbebrokendownasfollow(byindicativechronologicalorder):
ii)Develop a species-semantic segmentation network;
iii)Add an instance segmentation branch;
v)Combine information from aligned time-sequence of superspectral images, assess benefits;
vii)Produce an open-access, clean dataset for benchmarking andreproducibility.
(i) Data Formatting : several forest patches have been scanned and annotated by the French Forest Inventory. While extremely useful, the data is given at tree-level and needs to be re- formatted to be directly usable by a network. The information is sufficient : angle and distance from a the center of the point cloud, as well as species and morphological attributes. This easyfirststepwillconstituteavaluabledataexplorationphasefromtheperspectiveofaresearcher.
(ii)-(iii)SpeciesandInstanceSegmentation:Overthelastfewyears,anumberofneural networks have been developed for processing 3D data. We propose to use the powerful torch- points3d1open-source framework implementing many state-of-the-art methods for various ap- plications, including semantic segmentation and instance segmentation (and in our case, panop- ticsegmentation).Thiswillallowustoquicklyiterateandselectthemostrelevantarchitecture for ourproblem.
(iv) Morphological Regression : The annotated morphological descriptors can help the learning phase by qualifying the accuracy of theinstancesegments.Byimplementingdifferen- tiable proxy for the sought descriptors, we will be able to use these annotations in an end-to-end fashion to improve the instancesegmentation.
(v)-(vi) Multi-Modality : Multi-spectral and multi-temporal time series contains phenolo- gical information crucial to distinguish between plant species (Sainte-Fare-Garnot et al., 2020).Wepropose to align both Sentinel time-series and high-definition aerial imagery with our 3Ddata, and integrate this information to the point clouds. While the exact architecture of thisfusionschemeislefttodecide,ourgoalisforend-to-endjointlearning.
(vii) Dataset : The Dataset provided is atypical and challenging by its multi-modal and multi- task nature. We will package it into a open-access benchmark, relevant in the communities of remote sensing, computer vision, and machine learning.
Study Site and Data
The study site is situated in Southern Western France. A joint airborne multispectral andLiDARacquisitions(Automn2019)wereprocessed.Awinteracquisitionisscheduledinwinter 2020-2021. The spatial resolution is 20 cm GSD and a LiDAR point density equals 60 pts/m2. Variousscanangleacquisitionwereadded.31siteswith1840treeswereobservedandmeasured in field (height, crown diameter, stem diameter, density,specie,..).
—ThecandidateshouldhaveaPhDdegreeininformatics,remotesensing,computervision or machinelearning.
—Knowledge of deep learning methods and/or 3D point cloud processing is anasset.
—Mastery of Python is necessary, familiarity with PyTorch (preferred) or Tensorflow is a plus.
The candidate will work at UMR LaSTIG, UGE/IGN/EIVP Paris (STRUDEL team) in col- laboration with EA G&E and UMR BIOGECO in Bordeaux.
The IGN (Institut National de l’Information G´eographique et Foresti`ere), in Paris is the French National Mapping Agency. Jointky with EIVP and Universit´e Gustave Eifeel, the LaSTIG 2lab gathers full-time researchers on computer vision, machine learning, GIS and visualization of geographic information data. EA G&E has expertise on 3D point cloud processing and UMRBIOGECO represents the end-user of biodiversity metrics.
The post-doc position offers full health, unemployment and retirement benefits and compe- titive salary. The candidate will be given the opportunity to develop his/her research skills by contributing to 3D volumetric segmentation and semantic segmentation applied to geographic informationsciencesandforestry.Thefundingincludesatwo-yearpost-doctoralcontract and an operating fund (for travel, consumables, etc.). Salary per month : around 2300 euros (excluding taxes and with noaccommodation).
The position will start from October to December 2020.
Applications (resume+motivation letter+list ofpublicationsinasinglePDF)shouldbe sent before 10th October2020 to :