Abstract 1255P
Background
Artificial Intelligence (AI) and Natural Language Processing (NLP) advancements have led to sophisticated tools like GPT-4.0, allowing clinicians to explore its utility as a healthcare management support tool. Our study aimed to assess GPT-4's ability to suggest the definitive diagnosis and the most appropriate work-up to minimize unnecessary procedures.
Methods
We conducted a retrospective comparative analysis, extracting relevant clinical data from 10 cases published at NEJM after 2022 and inputting it into GPT-4 to generate diagnostic and workup recommendations. Primary endpoint: the ability to correctly identify the final diagnosis. Secondary endpoints: its ability to list the definitive diagnosis in the five most likely differential diagnoses and determine an adequate workup.
Results
The AI could not identify the definitive diagnosis in 2 out of the 10 cases (20% inaccuracy). Among the 8 cases correctly identified by the AI, 5 (63%) had the definitive diagnosis as the top differential diagnosis list. Regarding the suggested diagnostic tests and exams, requests for exams that would not aid in the patient's final diagnosis were made in 2 cases, representing 40% of the patients whose final diagnosis was not correctly identified by the AI. Moreover, the AI could not suggest adequate treatment for 7 cases (70%). Among them, the AI suggested inappropriate management for 2 cases, and the remaining 5 received incomplete or non-specific advice, such as chemotherapy, without specifying the best regimen.
Conclusions
Our study demonstrated GPT-4's potential as an academic support tool, although it cannot correctly identify the final diagnosis in 20% of the cases. There is also a limitation regarding the management suggested by AI. In cases where the main diagnostic hypothesis was incorrectly identified or not listed as the top differential diagnosis, the AI requested unnecessary additional diagnostic tests for 40% of the patients. Future research should focus on evaluating the performance of GPT-4 using a more extensive and diverse sample, incorporating prospective assessments, and investigating its ability to optimize diagnostic and therapeutic procedures to optimize healthcare utilization.
Clinical trial identification
Editorial acknowledgement
Legal entity responsible for the study
The authors.
Funding
Has not received any funding.
Disclosure
All authors have declared no conflicts of interest.
Resources from the same session
1203P - Role of tumor markers before or during chemotherapy for poorly differentiated neuroendocrine carcinomas of the digestive system: An exploratory analysis of JCOG1213
Presenter: Tomoyuki Satake
Session: Poster session 14
1204TiP - Iadademstat in combination with paclitaxel in relapsed/refractory small cell lung carcinoma (SCLC) and extrapulmonary high grade neuroendocrine carcinoma (NEC)
Presenter: Neel Belani
Session: Poster session 14
1212P - Predictive value of a near-term prediction model for severe irAEs in cancer treatment with ICIs
Presenter: Jun Zhao
Session: Poster session 14
1213P - HRD complete: A novel NGS assay for detecting homologous recombination repair (HRR) gene alterations in prostate cancer
Presenter: Xin Ye
Session: Poster session 14
1214P - A novel machine learning based method to detect homozygous deletion of homologous recombination repair (HRR) genes in prostate cancer
Presenter: Jianqing Wang
Session: Poster session 14
1215P - Comparative analysis of cfDNA liquid biopsy and tumor-based next-generation sequencing (NGS) approaches
Presenter: Anastasiya Yudina
Session: Poster session 14
1216P - A spectroscopic liquid biopsy for the earlier detection of multiple cancer types
Presenter: Matthew Baker
Session: Poster session 14
1217P - Clinical evaluation of a CE-IVD liquid biopsy pan cancer genomic profiling test
Presenter: Timothy Crook
Session: Poster session 14
1218P - Exploring cancer care pathways in seven European countries: Identifying obstacles and opportunities for the role of artificial intelligence
Presenter: Shereen Nabhani
Session: Poster session 14