19 avril 2018

Catégorie : Doctorant

Ph.D. Thesis Proposal: CausalSpatio-temporal Mathematical Morphology: Theory and Algorithms

*Scientific fields:* Applied mathematics and Computer Science – Interaction between nonlinear video sequence processing and human-vision inspired information processing.

*Supervision:* This Ph.D. Thesis will be carried out at the Center for Mathematical Morphology (CMM) at MINES ParisTech, in France. The Ph.D. Thesis will be supervised by Jesus Angulo, jesus.angulo@mines-paristech.fr.

*Application:* Candidates should send a CV, a cover letter and the grades obtained during the last two years.

*Knowledge and skill requirements:* The applicant must have a sound knowledge of applied mathematics (linear analysis, metric and Riemannian geometry, partial differential equations, probabilities and statistics, physical modeling, etc), together with a background in image processing and machine learning, validated by a Research Master in this subject. He/she will also have to have a skill in computer science, more particularly in Python and eventually in C/C++ programming.

A sound command of spoken and written English is an absolute requirement. In addition, an experience in a foreign laboratory would be an asset for this position.

Video sequences and similar time-indexed datasets are mainly processed either as 3D (homogenous product of space x time) images or as a set of independent 2D images, without taking into account the mathematical nature of the spatio-temporal space, going from the causality to the existence of a underlying motion vector field which modifies (locally) the metric. Even recent papers on convolutional neural networks (CNN) approaches for spatio-temporal video deep learning still focus on addressing both approaches [Xie et al., 2017]: “Is it important that we 3D convolve jointly over time and space? Or would it suffice to convolve over dimensions independently?”.

The fundamental reasons behind this limitation to genuinely tackle (space x time) are both theoretical and algorithmic.Indeed, especially in the case of real-time processing of video streams, the algorithms should be fast enough to follow the video rate and this was in the past generally incompatible with processing a buffer of previous frames. In the case of off-line processing of video sequences, the speed is not a major constraint; however it is preferred to see video sequences as (non-casual) 3D images since the theory and algorithms available in classic 3D image processing are widely available and can be applied without any change.

Classical scale-space theory is concerned with the mathematical modeling of front-end visual system behavior. In this context, inspired by measurement on the spatio-temporal responses of visual receptive fields and axiomatic formulation of scale-scale theory (i.e., invariance, semi-group structure, etc.), the scale-time kernels has been introduced to deal with real-time causal modeling [ter Haar Romeny et al., 2001]. This kind of theory has been extended to the case of spatio-temporal receptive fields and image data, in particular by considering relative motions between objects in the world and the observer, where a constant velocity translational motion can be modeled by a Galilean transformation [Lindeberg, 2013;2016]. Corresponding time-causal spatio-temporal kernels are related to a system of diffusion equations [Lindeberg, 2011]. Theoretical works on wavelets for physics have also considered the formulation of wavelets on affine (Galilean) groups space-time [Ali et al., 2014], which can be potentially used on video processing.

In the literature of mathematical morphology, there are a few works which have considered properly the formulation of spatio-temporal operators, in particular the idea of structuring elements following the optical flow [Laveau and Bernard, 2005] or sophisticated spatio-temporal structuring elements which decouple time/space connectivity [Luengo-Oroz et al., 2012]. However, despite those past contributions, the lack of a sound theoretical background and corresponding efficient algorithms involves that most of morphological methods nowadays considered for processing video sequences disregard the spatio-temporal phenomenology.

The goal of this thesis is just fill this gap by revisiting the main operators and algorithms of modern mathematical morphology for the case of functions and sets defined on the casual spatio-temporal domain.

- Set morphology on causal spatial-temporal poset: Set (or binary) morphology is based on the Minkowki sum and difference of sets, which uses the notion of set union and set intersection as supremum and infimum of a more abstract formulation of morphological operators on complete lattices. The extension to the causal spatial-temporal framework requires the formulation of counterpart for the supremum and infimum of casual partially-ordered set (poset) of events. This issue of ordering on space-time (or ordering on hyperboloid of Minkowski space-time) have been previously considered [J. Hilgert, 1991][Knuth and Bahrenyi, 2010] and it is also related to the Löwner partial order on Lorentz cone [Stott, 2016].
- Causal spatio-temporal morphological structuring functions and corresponding continuous models: Starting from the corresponding Hamilton-Jacobi PDE models of which are related to morphological operators or from the theory of non-linearization of standard scale-spaces [Angulo, 2016] [Schmidt and Weickert, 2017], the theory of multi-scale structuring function on the spacetime framework should include an axiomatic study of the semi-group properties, invariance to groups of transforms on video sequences, lipschitz regularization properties, etc. From a more practical viewpoint, the causal space-time structuring functions and structuring elements should be assessed on their interest on advanced morphological operators for object extraction and segmentation, typically, corresponding hit-or-miss transforms, granulometric decompositions, etc.
- Causal spatio-temporal connected and geodesic operators: Geodesic morphological operators are based on the notion of connectivity and reconstruction of connected components. The Riemannian geometry of the Minkowski Spacetime provides the metric and other ingredients required to formulate geodesic Riemanian operators [Angulo and Velasco-Forero, 2014] on this space, going from shortest paths, extraction of maxima/minima of a function to regularization by levelings and other connected filters.
- Casual spatio-temporal stochastic models of random sets: Mathematical morphology is intimately related to a family a random set models which are just fully characterized by morphological operators. Classical theory is formulated on the Euclidean framework, but there are some results of integral geometry on hyperbolic spaces [Santalo, 1968] which can be the starting point to deal with the counterpart of Boolean or Dead leaves models characterization on spatial-temporal framework. The extension of such models can be relevant to study dynamic microstructures or complex images of moving objects viewed as (space x time) stochastic textures.

Developing efficient time-casual and time-recursive morphological algorithms with requirements of speed, controlled complexity, etc. will need also to study during the thesis the best image representations (i.e., data structures such as tensors or graphs are the natural alternatives) which typically should incorporate also motion estimation.

- Computation of spatio-temporal morphological features and image descriptors from image streams in (near) real-time processing.
- Specific algorithms for embedded systems, in particular smart cameras.
- Analysis of video-sequences in different domains going from video-surveillance to time-lapse biomicroscopy, in particular for problems where the objects in the videos moving at very different speeds or in cases where the limit speed is known.
- Develop non-linear models for spatio-temporal receptive fields in biological vision systems and mathematical models related to the field of neurogeometry.
- Primitives for human-vision inspired video-CNN and other approaches in deep learning.

[Ali et al., 2014] S.T. Ali, J.P. Antoine, J.P. Gazeau. Wavelets Related to Affine Groups. In: Coherent States, Wavelets, and Their Generalizations. Theoretical and Mathematical Physics. Springer, New York, 2014.

[Angulo and Velasco-Forero, 2014] J. Angulo and S. Velasco-Forero. Riemannian mathematical morphology. Pattern Recognition Letters, Vol. 47, 93-101, 2014.

[Angulo 2016] J. Angulo. Generalised morphological image diffusion. Nonlinear Analysis: Theory, Methods & Applications, Vol. 134, 1-30, 2016.

[Hilgert, 1991] J. Hilger. The hyperboloid as ordered symmetric space. Seminar Sophus Lie, Vol.1, 135-141, 1991

[Knuth and Bahrenyi, 2010] K.H. Knurh, N. Bahrenyi. The Order-Theoretic Origin of Special Relativity. AIP Conference Proceedings 1305, 115-121 2011.

[Laveau and Bernard, 2005] N. Laveau, C. Bernard.Structuring Elements Following the Optical Flow. In: Ronse C., Najman L., Decencière E. (eds) Mathematical Morphology: 40 Years On. Computational Imaging and Vision, vol 30. Springer, 2005.

[Lindeberg, 2011] T. Lindeberg. Generalized Gaussian scale-space axiomatic comprising linear scale-space, affine scale-space and spatio-temporal scale-space. Journal of Mathematical Imaging and Vision 40(1): 36-81, 2011.

[Lindeberg, 2013] T. Lindeberg. Invariance of visual operations at the level of receptive fields. PLOS ONE, 8(7): e66990:1-33, 2013.

[Lindeberg, 2016] T. Lindeberg. Time-causal and time-recursive receptive fields. Journal of Mathematical Imaging and Vision 55(1): 50-88, 2016.

[Luengo-Oroz et al., 2012] M.A. Luengo-Oroz, D. Pastor, C. Castro, E. Faure, T. Savy, B. Lombardot, J. Rubio-Guivernau, L. Duloquin, M. Ledesma-Carbayo, P. Bourgine, N. Peyriéras, A. Santos. 3D+t Morphological Processing: Applications to Embryogenesis Image Analysis. IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3518-3530, 2012.

[Schmidt and Weickert, 2015] M. Schmidt, J. Weickert. The Morphological Equivalents of Relativistic and Alpha-Scale-Spaces. In: Aujol JF., Nikolova M., Papadakis N. (eds) Scale Space and Variational Methods in Computer Vision. SSVM 2015. Lecture Notes in Computer Science, vol 9087. Springer, 2015.

[Santalo, 1968] L. A.Santalo. Horospheres and convex bodies in hyperbolic space. Proc.Amer.Math. Soc., Vol. 19, 390–395, 1968.

[Stott, 2016] N. Stott. Maximal Lower Bounds in the Löwner order. arXiv: 1612.05664, 2016.

[ter Haar Romeny, 2001] B.M. ter Haar Romeny, L.M.J. Florack, M.Nielsen (2001) Scale-Time Kernels and Models. In: Kerckhove M. (eds) Scale-Space and Morphology in Computer Vision. Scale-Space 2001. LNSCS vol 2106. Springer, 2001.

[Xie et al., 2017] S. Xie, C. Sun, J. Huang, Z. Tu, K. Murphy. Rethinking Spatiotemporal Feature Learning For Video Understanding. arXiv:1712.04851, 2017.

(c) GdR 720 ISIS - CNRS - 2011-2018.