Multimodal Image Processing: from Conventional to Deep Learning Approach

Abstract

Multimodal image processing is one of the most popular image processing methods, and it is a method that solves problems by using multiple modal images together instead of using one single image. By using multiple modal images, multimodal image processing methods can solve more complex cases. This multimodal image processing approach has been applied in various ways, starting from image processing using conventional image processing algorithms to complex image analysis using deep learning techniques in recent years. In this study, we present two novel multimodal image processing methods that are more effective to analyze medical images. First, we present an algorithm to compensate for the motion problem that can occur in three-dimensional image acquisition of PA/US and show that the corrected images can be used for effective multispectral PA imaging. In this algorithm, the multi-wavelength PA/US images are used together to correct the motion, and the final spectral unmixing results are obtained. Second, we developed deep learning based models to segment and classify breast cancers. B-mode and SE-mode US images are utilized together to provide complementary information for improved analysis. Through these two cases, we will show how to process multimodal images using conventional and deep learning methods.