Investigate a new hybrid deep learning architecture for automatic generation of medical report. The project will consist in developing a multi-level multi attention (MLMA) architecture with the combination of CNN (extracting visual features from the original image), Long Short Term Memory (LSTM) and Bidirectional-LSTM for the generation of radiological report given chest x-ray images. The combination of context level visual attention and textual attention should ensure MLMA model to learn the syntactical and structural pattern, which should sequentially generate a plausible medical report. The work will be first concentrated on chest X-rays medical images for which image databases and associated reports are available on-line ; later on, a possible extension could be in the field of retinal disease, cardiology (from MRI inputs, new challenge for the image input!) or digital mammography through established partnerships and collaborations.
Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. arXiv preprint arXiv:1711.08195 (2017).
Krause, J., Johnson, J., Krishna, R., Fei-Fei, L.: A hierarchical approach for generating descriptive image paragraphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 317-325 (2017).
Wang, X., Peng, Y., Lu, L., Lu, Z., Summers, R.M.: Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9049-9058 (2018).
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International Conference on Machine Learning. pp. 2048-2057 (2015)
Application deadline Mai 15 - 2019.
Grant type: Bourse Ministére, around 1500 Euros/month for 3 years.
Starting date: Mid september 2019.
ImvIa Research Laboratory
Université de Bourgogne
(c) GdR 720 ISIS - CNRS - 2011-2018.