Ultrasound Image Generation and Modality Conversion Based on Deep Learning

doi:10.37015/AUDT.2023.230011

Abstract

Abstract:

Artificial intelligent (AI) based on deep learning has been used in medical imaging analysis for years. Improvements have been made in the diagnosis of various diseases with the help of deep learning. Multimodal medical imaging combines two or more imaging modalities, providing comprehensive diagnostic information of the diseases. However, some modality problems always exist in clinical practice. Recently, AI-based deep learning technologies have realized the modality conversion. Investigations on modality conversion have gradually been reported in order to acquire multimodal information. MRI images could be generated from CT images while ultrasound elastography could be generated from B mode ultrasonography. Continuous researches and development of new technologies around deep learning are still under investigation and provide huge clinical potentials in the future. The purpose of this review is to summarize an overview of the current applications and prospects of deep learning-based modality conversion of medical imaging.

Key words: Ultrasound image generation; Modality conversion; Deep learning

Shujun Xia, MD, Jianqiao Zhou, MD. Ultrasound Image Generation and Modality Conversion Based on Deep Learning. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(2): 136-139.

Figures/Tables 1

References 24

[1]	Baltrusaitis T, Ahuja C, Morency L P. Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell 2019; 41:423-443. doi: 10.1109/TPAMI.2018.2798607
[2]	Kieselmann J P, Fuller C D, Gurney-Champion O J, Oelfke U. Cross-modality deep learning: contouring of MRI data from annotated CT data only. Med Phys 2021; 48:1673-1684. doi: 10.1002/mp.v48.4
[3]	Mao L, Wu B, Wu X, Pei X, Xu X. Newest progresses of generation of synthetic CT based on deep learning in MRI-only radiotherapy. Chinese Journal of Medical Imaging Technology 2022; 38:619-623.
[4]	Lei Y, Harms J, Wang T, Liu Y, Shu H K, Jani A B, et al. MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks. Med Phys 2019; 46:3565-3581. doi: 10.1002/mp.13617 pmid: 31112304
[5]	Boulanger M, Nunes J C, Chourak H, Largent A, Tahri S, Acosta O, et al. Deep learning methods to generate synthetic CT from MRI in radiotherapy: a literature review. Phys Med 2021; 89:265-281. doi: 10.1016/j.ejmp.2021.07.027 pmid: 34474325
[6]	Meng X, Wu X, Peng Z, Xu X, Pei X. Auto-segmentation of organs-at-risk in abdominal CT after combining with synthetic-MRI information. Chinese Journal of Medical Physics 2022; 39:203-208.
[7]	Jin C B, Kim H, Liu M, Jung W, Joo S, Park E, et al. Deep CT to MR synthesis using paired and unpaired data. Sensors (Basel) 2019; 19:2361. doi: 10.3390/s19102361
[8]	Feng E, Qin P, Chai R, Zeng J, Wang Q, Meng Y, et al. MRI generated from CT for acute ischemic stroke combining radiomics and generative adversarial networks. IEEE J Biomed Health Inform 2022; 26:6047-6057. doi: 10.1109/JBHI.2022.3205961
[9]	Li W, Li Y, Qin W, Liang X, Xu J, Xiong J, et al. Magnetic resonance image (MRI) synthesis from brain computed tomography (CT) images based on deep learning methods for magnetic resonance (MR)-guided radiotherapy. Quant Imaging Med Surg 2020; 10:1223-1236. doi: 10.21037/qims
[10]	Pan K, Cheng P, Huang Z, Lin L, Tang X. Transformer-based T2-weighted MRI synthesis from T1-weighted images. Annu Int Conf IEEE Eng Med Biol Soc 2022;2022:5062-5065.
[11]	Kawahara D, Nagata Y. T1-weighted and T2-weighted MRI image synthesis with convolutional generative adversarial networks. Rep Pract Oncol Radiother 2021; 26:35-42. doi: 10.5603/RPOR.a2021.0005
[12]	Yang Q, Li N, Zhao Z, Fan X, Chang E I, Xu Y. MRI cross-modality image-to-image translation. Sci Rep 2020; 10:3753. doi: 10.1038/s41598-020-60520-6 pmid: 32111966
[13]	Dai X, Lei Y, Fu Y, Curran W J, Liu T, Mao H, et al. Multimodal MRI synthesis using unified generative adversarial networks. Med Phys 2020; 47:6343-6354. doi: 10.1002/mp.14539 pmid: 33053202
[14]	Cong W, Yang J, Liu Y, Wang Y. Fast and automatic ultrasound simulation from CT images. Comput Math Methods Med 2013; 2013: 327613.
[15]	Gjerald S U, Brekken R, Hergum T, D'Hooge J. Real-time ultrasound simulation using the GPU. IEEE Trans Ultrason Ferroelectr Freq Control 2012; 59:885-892. doi: 10.1109/TUFFC.2012.2273 pmid: 22622973
[16]	Satheesh B A, Thittai A K. A fast method for simulating ultrasound image from patient-specific CT data. Biomedical Signal Processing and Control 2019; 48:61-68. doi: 10.1016/j.bspc.2018.10.003
[17]	Cronin N J, Finni T, Seynnes O. Using deep learning to generate synthetic B-mode musculoskeletal ultrasound images. Comput Methods Programs Biomed 2020; 196:105583. doi: 10.1016/j.cmpb.2020.105583
[18]	Tom F, Sheet D. Simulating patho-realistic ultrasound images using deep generative networks with adversarial learning. in 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) 2018.
[19]	Wang R, Fang Z, Gu J, Guo Y, Zhou S, Wang Y, et al. High-resolution image reconstruction for portable ultrasound imaging devices. EURASIP Journal on Advances in Signal Processing 2019; 2019:56. doi: 10.1186/s13634-019-0649-x
[20]	Wu S, Gao Z, Liu Z, Luo J, Zhang H, Li S. Direct reconstruction of ultrasound elastography using an end-to-end deep neural network. in Medical Image Computing and Computer Assisted Intervention-MICCAI 2018 ; Cham: Springer International Publishing.
[21]	Kibria G, Rivaz H. GLUENet: ultrasound elastography using convolutional neural network: International Workshops, POCUS 2018, BIVPCS 2018, CuRIOUS 2018, and CPM 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16-20, 2018, Proceedings. 2018. p. 21-28.
[22]	Zhang Q, Zhao J, Long X, Luo Q, Wang R, Ding X, et al. AUE-Net: automated generation of ultrasound elastography using generative adversarial network. Diagnostics (Basel) 2022; 12:253.
[23]	Wildeboer R R, van Sloun R J G, Mannaerts C K, Moraes P H, Salomon G, Chammas M C, et al. Synthetic elastography using B-mode ultrasound through a deep fully convolutional neural network. IEEE Trans Ultrason Ferroelectr Freq Control 2020; 67: 2640-2648. doi: 10.1109/TUFFC.58
[24]	Yao Z, Luo T, Dong Y, Jia X, Deng Y, Wu G, et al. Virtual elastography ultrasound via generative adversarial network for breast cancer diagnosis. Nat Commun 2023; 14:788. doi: 10.1038/s41467-023-36102-1