Advanced analysis of medical image modality using generative and vision transformer model
*Hemanta Kumar BhuyanCorresponding authorhmb.bhuyan@gmail.comMuma College of Business University of South Florida 8350 N. Tamiami Trail SarasotaFlorida, FL 34243, U.S.A.0000-0002-9712-7280View full profile → , Bhuvan Unhelkarbunhelkar@usf.eduMuma College of Business University of South Florida 8350 N. Tamiami Trail SarasotaFlorida, 33620, USA0000-0003-1118-3837View full profile → , S. Siva Shankardrsivashankars@gmail.comDepartment of Computer Science and Engineering KG Reddy College of Engineering and TechnologyDepartment of Computer Science and Engineering KG Reddy College of Engineering and TechnologyHyderabad, Telangana, 500100, India0000-0002-7616-6088View full profile → , Prasun Chakrabartidrprasun.cse@gmail.comDepartment of Computer Science and Engineering Sir Padampat Singhania UniversityUdaipur, Rajasthan, 313601, lndia0000-0001-8062-4144View full profile →
* Corresponding author · click or hover a name for details
- Received:
- 05 Jun 2024
- Published Online:
- 04 Aug 2025
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JSMS-1409
- Pages:
- 1191–1222
Abstract
Keywords
Subject Classifications
References
[1] B. B. Thukral, “Problems and preferences in pediatric imaging,” Indian Journal of Radiology and Imaging, vol. 25, no. 4, pp. 359–364 (Oct. 2015).
[2] K. Krupa and M. Bekieśińska-Figatowska, “Artifacts in magnetic resonance imaging,” Polish Journal of Radiology, vol. 80, pp. 93–106 (Feb. 2015).
[3] A. Adam, A. Dixon, J. Gillard, C. Schaefer-Prokop, R. Grainger, and D. Allison, Grainger & Allison’s Diagnostic Radiology, Amsterdam, The Netherlands: Elsevier (2014).
[4] D. Ellison, Neuropathology: A Reference Text of CNS Pathology, Amsterdam, The Netherlands: Elsevier (2012).
[5] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Zhou, and L. Xing, “TransUNet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306 (2021).
[6] Y. Luo, H. Gong, J. Liu, Y. Liu, and H. Huang, “3D transformer-GAN for high-quality PET reconstruction,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2021, Cham, Switzerland: Springer, pp. 276–285 (2021).
[7] H. K. Bhuyan and V. K. Ravi, “An integrated framework with deep learning for segmentation and classification of cancer disease,” International Journal on Artificial Intelligence Tools, vol. 32, no. 02, Article no. 2340002 (2023).
[8] Y. Korkmaz, S. U. Dar, M. Yurt, M. Özbey, and T. Çukur, “Unsupervised MRI reconstruction via zero-shot learned adversarial transformers,” IEEE Transactions on Medical Imaging, early access, Jan. 27 (2022), doi: 10.1109/TMI.2022.3147426.
[9] H. K. Bhuyan, A. Vijayaraj, and V. K. Ravi, “Development of secrete images in image transferring system,” Multimedia Tools and Applications, vol. 82, no. 5, pp. 7529–7552 (2023).
[10] R. S. M. Patibandla and N. Veeranjaneyulu, “A SimRank based ensemble method for resolving challenges of partition clustering methods,” Journal of Scientific & Industrial Research, vol. 79, no. 4, pp. 323–327 (2022).
[11] A. Dureja and P. Pahwa, “Medical image retrieval for detecting pneumonia using binary classification with deep convolutional neural networks,” unpublished.
[12] C. Bowles, L. Chen, R. Guerrero, P. Bentley, D. Rueckert, and A. Hammers, “Pseudo-healthy image synthesis for white matter lesion segmentation,” in Simulation and Synthesis in Medical Imaging, Cham, Switzerland: Springer, pp. 87–96 (2016).
[13] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (Jun. 2016).
[14] K. Dhar, P. Bhattacharya, N. Kumar, and A. Mitra, “Abnormality detection in chest diseases using a convolutional neural network,” Journal of Information & Optimization Sciences, vol. 44, no. 1, pp. 97–111 (2023), doi: 10.47974/JIOS-1298.
[15] H. K. Bhuyan, A. Vijayaraj, and V. Ravi, “Diagnosis system for cancer disease using a single setting approach,” Multimedia Tools and Applications, Springer US, pp. 1–27 (2023).
[16] A. Torrado-Carvajal, M. Sanches, D. Pascau, J. C. Desco, and J. M. Santos, “Fast patch-based pseudo-CT synthesis from T1-weighted MR images for PET/MR attenuation correction in brain studies,” Journal of Nuclear Medicine, vol. 57, no. 1, pp. 136–143 (Jan. 2016).
[17] V. Sevetlidis, M. V. Giuffrida, and S. A. Tsaftaris, “Whole image synthesis using a deep encoder-decoder network,” in Simulation and Synthesis in Medical Imaging, Cham, Switzerland: Springer, pp. 127–137 (2016).
[18] D. J. Eckman, S. G. Henderson, and S. Shashaani, “Diagnostic tools for evaluating and comparing simulation-optimization algorithms,” INFORMS Journal on Computing, vol. 35, no. 2, pp. 350–367 (2023).
[19] H. K. Bhuyan, V. Ravi, B. Brahma, and N. K. Kamila, “Disease analysis using machine learning approaches in healthcare system,” Health and Technology, vol. 12, no. 5, pp. 987–1005 (2022).
[20] S. U. H. Dar, M. Yurt, L. Karacan, A. Erdem, E. Erdem, and T. Çukur, “Image synthesis in multi-contrast MRI with conditional generative adversarial networks,” IEEE Transactions on Medical Imaging, vol. 38, no. 10, pp. 2375–2388 (Oct. 2019).
[21] N. Cordier, H. Delingette, M. Le, and N. Ayache, “Extended modality propagation: Image synthesis of pathological cases,” IEEE Transactions on Medical Imaging, vol. 35, no. 12, pp. 2598–2608 (Dec. 2016).
[22] H. K. Bhuyan, C. Chakraborty, Y. Shelke, and S. K. Pani, “COVID-19 diagnosis system by deep learning approaches,” Expert Systems, vol. 39, no. 3, p. e12776 (2022).
[23] K. Bahrami, F. Shi, X. Zong, H. W. Shin, H. An, and D. Shen, “Reconstruction of 7T-like images from 3T MRI,” IEEE Transactions on Medical Imaging, vol. 35, no. 9, pp. 2085–2097 (Sep. 2016).
[24] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 27, pp. 2672–2680 (2014).
[25] M. Frid-Adar, I. Diamant, E. Klang, M. Amitai, J. Goldberger, and H. Greenspan, “GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification,” Neurocomputing, vol. 321, pp. 321–331 (Dec. 2018).
[26] T. Zhou, H. Fu, G. Chen, J. Shen, and L. Shao, “Hi-Net: Hybrid-fusion network for multi-modal MR image synthesis,” IEEE Transactions on Medical Imaging, vol. 39, no. 9, pp. 2772–2781 (Sep. 2020).
[27] R. S. M. L. Patibandla, A. Yaswanth, and S. I. Hussani, “Water-body segmentation from remote sensing satellite images utilizing hierarchical and contour-based multi-scale features,” in Mobile Radio Communications and 5G Networks, Lecture Notes in Networks and Systems, vol. 588, Singapore: Springer, pp. 245–253 (2023).
[28] H. K. Bhuyan, M. Saikiran, M. Tripathy, and V. Ravi, “Wide-ranging approach-based feature selection for classification,” Multimedia Tools and Applications, vol. 82, no. 15, pp. 23277–23304 (Jun. 2023).
[29] A. Sharma and G. Hamarneh, “Missing MRI pulse sequence synthesis using multi-modal generative adversarial network,” IEEE Transactions on Medical Imaging, vol. 39, no. 4, pp. 1170–1183 (Apr. 2020).
[30] G. Wang, W. Li, M. A. Zuluaga, B. Dou, A. Vercauteren, and T. Fletcher, “Synthesize high-quality multi-contrast magnetic resonance imaging from multi-echo acquisition using multi-task deep generative model,” IEEE Transactions on Medical Imaging, vol. 39, no. 10, pp. 3089–3099 (Oct. 2020).
[31] Y. Hiasa, Y. Otake, M. Takao, K. Matsuoka, and K. Sato, “Cross-modality image synthesis from unpaired data using CycleGAN: Effects of gradient consistency loss and training data size,” in Simulation and Synthesis in Medical Imaging, Cham, Switzerland: Springer, pp. 31–41 (2018).
[32] C.-B. Jin, J.-H. Kim, D.-Y. Lee, and J.-M. Kim, “Deep CT to MR synthesis using paired and unpaired data,” Sensors, vol. 19, no. 10, p. 2361 (May 2019).
[33] Y. Yang, K. Zhang, and Y. Fan, “sDTM: A supervised Bayesian deep topic model for text analytics,” Information Systems Research, vol. 34, no. 1, pp. 137–156 (2022).
[34] H. K. Bhuyan and V. K. Ravi, “Analysis of sub-feature for classification in data mining,” IEEE Transactions on Engineering Management, vol. 70, no. 8, pp. 2732–2746 (2023).
[35] H. Liang and Y. Xue, “Save face or save life: Physicians’ dilemma in using clinical decision support systems,” Information Systems Research, vol. 33, no. 2, pp. 737–758 (2023).
[36] H. K. Bhuyan, V. Ravi, and M. S. Yadav, “Multi-objective optimization-based privacy in data mining,” Cluster Computing, vol. 25, no. 6, pp. 4275–4287 (2022).
[37] M. Li, W. Hsu, X. Xie, J. Cong, and W. Gao, “SACNN: Self-attention convolutional neural network for low-dose CT denoising with self-supervised perceptual loss network,” IEEE Transactions on Medical Imaging, vol. 39, no. 7, pp. 2289–2301 (Jul. 2020).
[38] X. Zhang, Y. Wang, H. Wang, and C. Liu, “PTNet: A high-resolution infant MRI synthesizer based on transformer,” arXiv preprint arXiv:2105.13993 (2021).
[39] Y. Su, D. Jia, Y. Shen, and L. Wang, “Single-channel blind image separation based on transformer-guided GAN,” Sensors, vol. 23, no. 10, p. 4638 (2023).
[40] O. Dalmaz, M. Yurt, and T. Çukur, “ResViT: Residual vision transformers for multimodal medical image synthesis,” IEEE Transactions on Medical Imaging, vol. 41, no. 10, pp. 2598–2614 (Oct. 2022).
[41] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox, “On the information bottleneck theory of deep learning,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, no. 12, p. 124020 (2019).
[42] B. C. Geiger, “On information plane analyses of neural network classifiers—A review,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 12, pp. 7039–7051 (Dec. 2022).
[43] R. Shwartz-Ziv, A. Painsky, and N. Tishby, “Representation compression and generalization in deep neural networks,” OpenReview (2019). [Online]. Available: https://openreview.net/forum?id=SkeL6sCqK7
[44] H. Hafez-Kolahi, S. Kasaei, and M. Soleymani-Baghshah, “Sample complexity of classification with compressed input,” Neurocomputing, vol. 415, pp. 286–294 (2020).
[45] K. Kawaguchi, Z. Deng, X. Ji, and J. Huang, “How does information bottleneck help deep learning?,” in Proc. 40th Int. Conf. Mach. Learn. (ICML), Honolulu, Hawaii, USA, PMLR 202, pp. 1–48 (2023). [Online]. Available: https://arxiv.org/abs/2305.18887
[46] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1125–1134 (2017).
[47] Z. Zhou, R. Bai, J. Munasinghe, L. Nie, and X. Chen, “T1-T2 dual-modal magnetic resonance imaging: From molecular basis to contrast agents,” ACS Nano, vol. 11, no. 6, pp. 5227–5232 (Jun. 2017), doi: 10.1021/acsnano.7b03075.
[48] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, et al., “The multimodal brain tumor image segmentation benchmark (BRATS),” IEEE Transactions on Medical Imaging, vol. 34, no. 10, pp. 1993–2024 (Oct. 2014).
[49] T. Nyholm, E. Nyberg, L. Karlsson, and A. Carlsson Tedgren, “MR and CT data with multiobserver delineations of organs in the pelvic area—Part of the gold atlas project,” Medical Physics, vol. 45, no. 3, pp. 1295–1300 (Mar. 2018).
[50] M. Jenkinson and S. Smith, “A global optimisation method for robust affine registration of brain images,” Medical Image Analysis, vol. 5, no. 2, pp. 143–156 (Jun. 2001).
[51] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, et al., “Attention U-Net: Learning where to look for the pancreas,” arXiv preprint arXiv:1804.03999 (2018).
[52] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in Proc. Int. Conf. Machine Learning (ICML), vol. 97, pp. 7354–7363 (2019).
[53] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local Nash equilibrium,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 6629–6640 (2017).
[54] R. Durall, M. Keuper, and J. Keuper, “Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce texture,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), pp. 7880–7889 (2020).
[55] L.-H. Chen, C. G. Bampis, Z. Li, C. Chen, and A. C. Bovik, “Convolutional block design for learned fractional downsampling,” arXiv preprint arXiv:2105.09999 (2021).
[56] N. Kodali, J. Abernethy, J. Hays, and Z. Kira, “On convergence and stability of GANs,” arXiv preprint arXiv:1705.07215 (2017).
[57] Y. Luo, Y. Gao, L. Zhang, Z. Zhang, M. Jiang, and S. Hu, “Edge-preserving MRI image synthesis via adversarial network with iterative multi-scale fusion,” Neurocomputing, vol. 452, pp. 63–77 (Sep. 2021).
[58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 1–11 (2017).
