Vous êtes ici : Accueil » Kiosque » Annonce

Identification

Identifiant: 
Mot de passe : 

Mot de passe oublié ?
Détails d'identification oubliés ?

Annonce

2 octobre 2018

PhD: Registration of Heterogenous Data for Urban Modeling


Catégorie : Doctorant


The Building Indoor & Outdoor Modelling (BIOM) project aims at automatic, simultaneous indoor and outdoor modelling of buildings from images and dense point clouds. We want to achieve a complete, geometrically accurate, semantically annotated but nonetheless lean 3D CAD representation of buildings and objects they contain in the form of a Building Information Models (BIM) that will help manage buildings in all their life cycle (renovation, simulation, deconstruction).

The aim of this PhD is to address all registration issues faced by the BIOM projetct, as a major objective of the project is exploiting jointly data from various sensors (images/LiDAR) and viewpoints (terrestrial indoor/outdoor, aerial). Four key objectives for the BIOM project have been identified:

1. Registration of outdoor heteregeneous data. 2. Registration of indoor data.
3. Indoor/outdoor landmark extraction.
4. Global indoor/outdoor registration.

A more general scientific problem with important applications concerns the registration of images to 3D data, so that we can add another objective:

5. Image/LiDAR registration.

 

Registration of Heterogenous Data for Urban Modeling

 

PhD proposal

1 Context

The Building Indoor & Outdoor Modelling (BIOM) project aims at automatic, simultaneous indoor and outdoor modelling of buildings from images and dense point clouds. We want to achieve a complete, geometrically accurate, semanti- cally annotated but nonetheless lean 3D CAD representation of buildings and objects they contain in the form of a Building Information Models (BIM) that will help manage buildings in all their life cycle (renovation, simulation, decon- struction). We view indoor and outdoor building modelling as a joint process where both worlds fruitfully cooperate and benefit one another both in terms of semantics and geometry. The hope is that this holistic scene understanding and reconstruction approach will lead to more complete, correct, and geometrically accurate building models.

The first challenge will be to accommodate for heterogeneous data as full building modeling calls for data acquisition inside and outside the building but also from an aerial point of view to model roof. The BIOM project will also aim at exploiting the complementarity of image and LiDAR data. Another challenge is coping with incomplete data due to occlusions by furniture inside and urban and mobile objects outside. Last but not least, BIOM aims at modeling a large variety of architectural styles, different interior scene layouts, and a high amount of different objects that may be contained within the scene.

State-of-the-art approaches treat outdoor and indoor worlds separately: most indoor reconstruction approaches focus on detailed modelling of single rooms whereas only very few have dealt with 3D modelling of complete floors (under Manhattan world assumptions). To the best of our knowledge, no works have been proposed, yet, that model buildings outdoor and indoor simultaneously within one single comprehensive framework.

The BIOM project is funded by ANR, the French national research funding agency.

 

2 Objectives

The aim of this PhD is to address all registration issues faced by the BIOM projetct, as a major objective of the project is exploiting jointly data from vari- ous sensors (images/LiDAR) and viewpoints (terrestrial indoor/outdoor, aerial). Four key objectives for the BIOM project have been identified:

1. Registration of outdoor heteregeneous data. 2. Registration of indoor data.
3. Indoor/outdoor landmark extraction.
4. Global indoor/outdoor registration.

A more general scientific problem with important applications concerns the registration of images to 3D data, so that we can add another objective:

5. Image/LiDAR registration.

2.1 Registration of outdoor heteregeneous data

While specific landmarks that can be detected, matched and located with a high- enough confidence are useful, we plan to also use denser features that are mostly invariant to changes of scale and viewpoints, i.e., straight lines. A multi-scale approach is however required for a stable line detection, especially in the context of registration [9]. Current line segment description and matching techniques [16] will however have to be adapted to robustly cover the case of significant changes of viewpoints and scales. As for relating pictures to LiDAR data, in addition to mutual information measures and cleaner reconstructed views of the point cloud using meshing techniques (to take occultation into account), we will also study the detection edge lines in points clouds (possibly via plane intersection) and the correspondence with lines detected in images. A promising registration technique based on a Structure-from-Motion (SfM) approach has recently been developed, that meaningfully associate in the same framework point and line features when available [10, 11]. We need to extend it to a full graph of views, possibly taking into account outliers and smoothing residual errors on the whole graph. We also need to extend it regarding scalability, to address the case of buildings, and possibly neighborhoods and whole cities.

2.2 Registration of indoor data

Indoor registration of photographies is a hard problem, mainly due to ubiqui- tous areas with little or no texture, and abrupt viewpoint changes when moving from one room to another. Other issues include degenerate cases for 3D anal- ysis when moving forward, typically along a corridor and very small baselines when picturing a room from the room center. We plan to extend preliminary work on these issues based on line detection, which are robust to textureless environment and wide viewpoint changes [10, 11, 5]. Indoor registration of laser scans is a rather well studied problem. However, in case of angled, narrow room geometries and small overlap of point clouds call for additional constraints, we will augment indoor registration with soft priors like co-planarity of walls, for example. Registered indoor data can be further consolidated using geometric [3, 1] and semantic Choi2013 prior knowledge.

2.3 Indoor/outdoor landmark extraction

Our landmark detection will rely on a study of geometric and photometric cues that suitable for a robust and accurate registration of a building outdoor and indoor. The semantic labelling of both indoor and outdoor will also be exploited to gather as many object-based cues as possible. These cues will need to have their location and geometry estimated as accurately as possible as precise co- registration is required for final joint labelling and reconstruction. First, we will investigate the possible pertinent landmarks that can be extracted from both indoor and outdoor. Three kinds of landmarks are conceivable as already described in a large body of literature [4]: (i) direct dense point cloud registra- tion based on the common labels obtained by other work packages within the project, or on more high-level 3D dense features [4]; (ii) primitive-based regis- tration, stemming from an analysis of surfaces (e.g., planes for walls, ground, and ceilings, [8]), but also of corners, lines and rectangles; and (iii) sparse key- point/landmark matching that can be mid- or high-level features (e.g., from corners [13] and surface descriptors robust to noise such as 3D SURF [6] to direct window detection). The specificity within the BIOM project is that fea- tures should be the same for indoor and outdoor while point cloud acquisition protocols, surface samplings, densities, and noise are rather different and event fluctuating w.r.t. occlusion and furniture. This issue started to be investigated in [12] but was limited to outdoor point clouds. Second, we will estimate the location, geometry and possibly other features associated to the detected land- mark to enable their matching, and the subsequent global registration. Regular- ity constraints will be derived for other BIOM sources of information instead of being established by a priori knowledge such as in [15]. We will also formulate recommendations regarding data acquisition to facilitate indoor/outdoor regis- tration, e.g., adding scans on door thresholds or near to windows (inside and outside), to capture cues from “the other side” and to improve feature detection accuracy and robustness, as well as to disambiguate matching.

2.4 Global indoor/outdoor registration

Outdoor and indoor landmarks shall be robustly matched, then used in a global registration framework to ensure the consistency of indoor with outdoor. Some of the classes of interest for the semantic labelling step are of high relevance for this and can serve as relevant landmarks for registration and reconstruction. For instance, walls, floors, ceilings will serve for coplanarity constraints, doors and windows for additional constraints in wall planes, and windows as premier cues for indoor/outdoor registration. A number of issues will be studied. We will investigate if it is advantageous to first register all indoor data (possibly including derived information : meshes, texture, semantic labels) and similarly all outdoor data separately, or if one shall better register all data globally at once, possible depending on the location accuracy of the various kinds of de- tected primitives and features. To go beyond rigid alignment, another key issue concerns the relative flexibility in the location of landmarks, possibly depending on their detection, to allow a deformable fitting of composite indoor and out- door data. Yet another issue concerns the optimization of this fitting: iterative refinement between the indoor and outdoor models shall be formulated in a global energy function where rich prior knowledge and indoor/outdoor interde- pendencies are for instance formulated with graphical models and higher-order potentials [14].

2.5 Image/LiDAR registration

The problem of registration of imagery to Lidar data is of general interest, not restricted to the BIOM project. Again, higher level primitives could be extracted, such as line segments, in order to be matched. Most literature is concerned with aerial imagery and LiDAR. Some use the maximization of the mutual information [7]. A few articles are concerned with terrestrial outdoor scenes, in the context of mobile mapping [2].

3 Profile of the candidate

• Student in a research oriented masters degree.
• Specialization in computer vision, image processing or optimization.• Good skills in C++ language.
• Strong taste for scientific research.

4 Environment

The PhD would begin during Fall 2018. The contacts and advisors for this PhD are:

Pascal MONASSE, Imagine team, Phone +33164152176, email monasse@imagine.enpc.fr

Bruno VALLET, MATIS team, +33143988081, email bruno.vallet@ign.fr

Pascal MONASSE is a full-time researcher at the Imagine research group of École des Ponts ParisTech (Marne-la-Vall ́ee, France, at 20 minutes from Paris center by train), specialized in computer vision, machine learning and optimiza- tion. Bruno VALLET is a full-time researcher of the MATIS team and coordinator of the BIOM project. The MATIS team depends on the research unit in geo-information science of the French mapping agency (IGN), specializing in computer science applied to photogrammetry, computer vision and remote sensing.

References

[1] Cabral, R., and Furukawa, Y. Piecewise planar and compact floorplan reconstruction from images. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on (2014), IEEE, pp. 628–635.
[2] Cui, T., Ji, S., Shan, J., Gong, J., and Liu, K. Line-based registration of panoramic images and LiDAR point clouds for mobile mapping. Sensors 17, 70 (2017).
[3] Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski, R. Recon- structing building interiors from images. In Computer Vision, 2009 IEEE 12th International Conference on (2009), IEEE, pp. 80–87.
[4] Gressin, A., Mallet, C., Demantké, J., and David, N. Towards 3d LiDAR point cloud registration improvement using optimal neighbor- hood knowledge. ISPRS journal of photogrammetry and remote sensing 79(2013), 240–251.
[5] Ikehata, S., Boyadzhiev, I., Shan, Q., and Furukawa, Y. Panoramic structure from motion via geometric relationship detection. arXiv preprint arXiv:1612.01256 (2016).
[6] Knopp, J., Prasad, M., Willems, G., Timofte, R., and Van Gool, L. Hough transform and 3d surf for robust three dimensional classification. In European Conference on Computer Vision (2010), Springer, pp. 589–602.
[7] Mastin, A., Kepner, J., and Fisher, J. Automatic registration of LiDAR and optical images of urban scenes. In CVPR (2009), Institute of Electrical and Electronics Engineers.
[8] Oesau, S., Lafarge, F., and Alliez, P. Indoor scene reconstruction using feature sensitive primitive extraction and graph-cut. ISPRS Journal of Photogrammetry and Remote Sensing 90 (2014), 68–82.
[9] Salaün, Y., Marlet, R., and Monasse, P. Multiscale line segment detector for robust and accurate SfM. In Pattern Recognition (ICPR), 2016 23rd International Conference on (2016), IEEE, pp. 2000–2005.
[10] Salaün, Y., Marlet, R., and Monasse, P. Robust and accurate line- and/or point-based pose estimation without manhattan assumptions. InEuropean Conference on Computer Vision (2016), Springer, pp. 801–818.
[11] Salau ̈n, Y., Marlet, R., and Monasse, P. Line-based robust SfM with little image overlap. In 3D Vision (3DV), 2017 International Conference on (2017), IEEE, pp. 195–204.
[12] Shan, Q., Wu, C., Curless, B., Furukawa, Y., Hernandez, C., and Seitz, S. M. Accurate geo-registration by ground-to-aerial image matching. In 3D Vision (3DV), 2014 2nd International Conference on (2014), vol. 1, IEEE, pp. 525–532.
[13] Theiler, P. W., Wegner, J. D., and Schindler, K. Keypoint-based 4-points congruent sets–automated marker-less registration of laser scans. ISPRS journal of photogrammetry and remote sensing 96 (2014), 149–163.
[14] Wegner, J. D., Montoya-Zegarra, J. A., and Schindler, K. A higher-order crf model for road network extraction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013), pp. 1698–1705.
[15] Xiao, J., and Furukawa, Y. Reconstructing the worlds museums. In- ternational journal of computer vision 110, 3 (2014), 243–258
[16] Zhang, L., and Koch, R. An efficient and robust line segment match- ing approach based on lbd descriptor and pairwise geometric consistency. Journal of Visual Communication and Image Representation 24, 7 (2013), 794–805.

 

http://imagine.enpc.fr/positions/theses2018/PhD_WP1_BIOM.pdf

 

Dans cette rubrique

(c) GdR 720 ISIS - CNRS - 2011-2018.