Performance of ChatGPT and Radiology Residents on Ultrasonography Board-Style Questions

doi:10.37015/AUDT.2024.240002

Abstract

Abstract:

Objective: This study aims to assess the performance of the Chat Generative Pre-Trained Transformer (ChatGPT), specifically versions GPT-3.5 and GPT-4, on ultrasonography board-style questions, and subsequently compare it with the performance of third-year radiology residents on the identical set of questions.
Methods: The study, conducted from May 19 to May 30, 2023, utilized a selection of 134 multiple-choice questions sourced from a commercial question bank for American Registry for Diagnostic Medical Sonography (ARDMS) examinations and imported into the ChatGPT model (encompassing GPT-3.5 and GPT-4 versions). ChatGPT’s responses were evaluated overall, by topic, and by GPT version. An identical question set was assigned to three third-year radiology residents, enabling a direct comparison of performances with ChatGPT.
Results: GPT-4 correctly responded to 82.1% of questions (110 of 134), significantly surpassing the performance of GPT-3.5 (P = 0.003), which correctly answered 66.4% of questions (89 of 134). Although GPT-3.5’s performance was statistically indistinguishable from the average performance of the radiology residents (66.7%, 89.3 of 134) (P = 0.969), there was a notable difference in the accuracy in question-answering accuracy between GPT-4 and the residents (P = 0.004).
Conclusions: ChatGPT demonstrated significant competency in responding to ultrasonography board-style questions, with the GPT-4 version markedly surpassing both its predecessor GPT-3.5 and the radiology residents.

Key words: Artificial intelligence; Ultrasonography; Accuracy; Medical education

Xu Jiale, Xia Shujun, Hua Qing, Mei Zihan, Hou Yiqing, Wei Minyan, Lai Limei, Yang Yixuan, Zhou Jianqiao. Performance of ChatGPT and Radiology Residents on Ultrasonography Board-Style Questions. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(4): 250-254.

Figures/Tables 3

Figure 1

Table 1

Table 2

References 9

[1]	OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt/, Nov 30, 2022. Accessed May 19, 2023.
[2]	Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH. Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology 2023; 307:e230424.
[3]	Mihalache A, Popovic MM, Muni RH. Performance of an artificial intelligence Chatbot in ophthalmic knowledge assessment. JAMA Ophthalmol 2023; 141:589-597. doi: 10.1001/jamaophthalmol.2023.1144 pmid: 37103928
[4]	Adams LC, Truhn D, Busch F, Karder A, Niehues SM, Makowski MR, et al. Leveraging GPT-4 for post hoc transformation of free-text radiology reports into structured reporting: A multilingual feasibility study. Radiology 2023; 307:e230725.
[5]	Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a radiology Board-style examination: Insights into current strengths and limitations. Radiology 2023; 307:e230582.
[6]	OpenAI. GPT-4 Technical Report. https://ui.adsabs.harvard.edu/abs/2023arXiv230308774O May 8, 2023. Accessed May 10, 2023.
[7]	Bhayana R, Bleakney RR, Krishna S. GPT-4 in radiology: Improvements in advanced reasoning. Radiology 2023; 307:e230987.
[8]	Ultrasound Registry Review Question Bank. https://www.prepry.com/ Accessed May 19, 2023.
[9]	Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, et al. ChatGPT and other large language models are double-edged swords. Radiology 2023; 307:e230163.

Question topics	Number of questions	Number of correct answers		P value
Question topics	Number of questions	GPT-3.5	GPT-4	P value
Physics	20	17(85)	18(90)	> 0.99
Clinic	114	72(63.2)	92(80.7)	0.003
Vascular technology	20	11(55)	19(95)	0.004
Abdomen	20	10(50)	15(75)	0.102
Obstetrics and gynecology	20	15(75)	18(90)	0.405
Pediatric sonography	20	10(50)	12(60)	0.525
Breast	20	14(70)	17(85)	0.449
Adult echocardiography	14	12(85.7)	11(78.6)	>0.99
Total	134	89(66.4)	110(82.1)	0.003

Question topics	Number of questions	Number of correct answers
Question topics	Number of questions	Resident A	Resident B	Resident C
Physics	20	14(70)	13(65)	12(60)
Clinic	114	72(63.2)	80(70.2)	77(67.5)
Vascular technology	20	14(70)	16(80)	15(75)
Abdomen	20	13(65)	12(60)	16(80)
Obstetrics and gynecology	20	15(75)	16(80)	15(75)
Pediatric sonography	20	9(45)	10(50)	9(45)
Breast	20	15(75)	13(65)	13(65)
Adult echocardiography	14	6(42.8)	13(92.8)	9(64.3)
Total	134	86(64.2)	93(69.4)	89(66.4)

[1]	Shi Ming-Li, Liu Guo-Hui, Bao Fu-Xing. Advances in the Application of Ultrasound for Evaluating Placental Function in Fetal Growth Restriction (FGR) Pregnancy [J]. Advanced Ultrasound in Diagnosis and Therapy, 2026, 10(1): 42-50.
[2]	Hu Xuelin, Zhu Ye, Zhang Zisang, Quan Yuanting, Chen Wenwen, Chen Leichong, Xu Guangyu, Qin Luning, Xie Mingxing, Zhang Li. Explainable Artificial Intelligence in Echocardiography [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(4): 409-425.
[3]	Li Yanran, Cui Yuanjie, Wu Qingqing, Zhang Na. Current Applications of Artificial Intelligence in Obstetric Ultrasound [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(4): 449-456.
[4]	Yu Xiao jie, Song Zheng lai, Chang Xue yong, Yu Jie, Liang Ping. Artificial Intelligence in Ultrasound Diagnosis of Liver Nodules: A Comprehensive Review of B-Mode and Contrast-enhanced Applications [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(4): 326-346.
[5]	Zhang Xiaoqian, Zhang Jingwen, Dong Yijie, Zhou Jianqiao. Research Progress and Clinical Translation of Photoacoustic–ultrasound Fusion Imaging in Breast Cancer Diagnosis and Therapy [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(4): 467-482.
[6]	Jin Tong, Yu Xiaohu, Ai Zheng, Guo Hongcheng. Artificial Intelligence in Ultrasound Imaging: A Review of Progress from Machine Learning to Large Language Model [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(4): 483-496.
[7]	Zheng Hairong, Meng Long, Li Fei, Niu Lili, Qiu Weibao, Ma Teng, Liu Chengbo, Zhu Xuefeng, Wan Liwen, Cai Feiyan. Advance in Ultrasound Super-resolution Imaging, Cell Manipulation and Inter-brain Communication [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(4): 307-325.
[8]	Zhang Rong, Xie Liting, Jin Qijing, Zhang Chengyue, Guo Teng, Zhao Qiyu, Jiang Tian’an. Research Progress on the Application of Irreversible Electroporation Ablation in Cancers [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(3): 229-244.
[9]	Elkouahy Fatima Ezzahra, Bennis Ahmed, Merke Nicolas, Ouahid Hajar, Malali Hamid El, Taleb Lhoucine Ben, Mouhsen Azeddine. Advanced Diagnosis of Aortic Stenosis Disease Based on Ultrasound Images: A Novel Artificial Intelligence Approach [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(3): 298-306.
[10]	Zhang Minyu, Jia Baocheng, Huang Liuming. Imaging Diagnosis of Neonatal Umbilical Arteriovenous Malformation Complicated with Portosystemic Shunt [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(2): 224-227.
[11]	An Zichen, Li Fan. Advancements in the Application of Convolutional Neural Networks in Ultrasound Imaging for Breast Cancer Diagnosis and Treatment [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(1): 21-31.
[12]	Li Tiantian, Zhu Miao, Shen Dejuan, Qian Xiaoqin. The Diagnostic Pitfall of Lymphadenopathy: Ultrasonic Imaging Findings in a Case of Cat Scratch Lymphadenitis [J]. Advanced Ultrasound in Diagnosis and Therapy, 2025, 9(1): 92-95.
[13]	Ge Xifeng, Liu Wenzheng, Chen Wen, Mei Fang, Cui Ligang. Gastroduodenal Intussusception Due to a Gastric Gastrointestinal Stromal Tumor in Adult from Sonographer's Perspective [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(4): 255-258.
[14]	Shanqing Li, MM, Rong Hu, MM, Xijing Liu, MD, Fan Yang, MD. Update on the Genetics and Prenatal Ultrasound Features of Williams-Beuren Syndrome [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(3): 79-85.
[15]	Guiwu Chen, MS, Zhizhong He, BS, Wenqin Liu, MS, Junjun Chen, BS, Xiaomin Liao, MS, Yuhuan Xie, BS. Multimodal Sonographic Findings of Embryonal Carcinoma in the Testis: A Case Report and Literature Review [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(2): 74-77.