Oops, you're using an old version of your browser so some of the features on this page may not be displaying properly.

MINIMAL Requirements: Google Chrome 24+Mozilla Firefox 20+Internet Explorer 11Opera 15–18Apple Safari 7SeaMonkey 2.15-2.23

Poster session 14

1255P - Evaluating GPT-4 as an academic support tool for clinicians: A comparative analysis of case records from the literature

Date

21 Oct 2023

Session

Poster session 14

Topics

Cancer Intelligence (eHealth, Telehealth Technology, BIG Data);  Cancer Diagnostics

Tumour Site

Presenters

Marcos Aurelio Fonseca Magalhaes Filho

Citation

Annals of Oncology (2023) 34 (suppl_2): S711-S731. 10.1016/S0923-7534(23)01942-7

Authors

M.A. Fonseca Magalhaes Filho1, P.N. Aguiar Junior2, B.L. Fabre1, F. Marques3, B. Gutierres4, W. Nassib William Junior5, A. Del Giglio2

Author affiliations

  • 1 Clinical Oncology, Beneficencia Portuguesa, 01323001 - Sao Paulo/BR
  • 2 Clinical Oncology Dept., FMABC - Centro Universitário Saúde ABC - Faculdade de Medicina, 09060-650 - Santo Andre/BR
  • 3 Pneumology, Beneficencia Portuguesa de Sao Paulo, 01323-001 - Sao Paulo/BR
  • 4 Oncology, USP - Universidade de Sao Paulo, 05508-220 - Sao Paulo/BR
  • 5 Oncology Center, MD Anderson, Houston/US

Resources

Login to get immediate access to this content.

If you do not have an ESMO account, please create one for free.

Abstract 1255P

Background

Artificial Intelligence (AI) and Natural Language Processing (NLP) advancements have led to sophisticated tools like GPT-4.0, allowing clinicians to explore its utility as a healthcare management support tool. Our study aimed to assess GPT-4's ability to suggest the definitive diagnosis and the most appropriate work-up to minimize unnecessary procedures.

Methods

We conducted a retrospective comparative analysis, extracting relevant clinical data from 10 cases published at NEJM after 2022 and inputting it into GPT-4 to generate diagnostic and workup recommendations. Primary endpoint: the ability to correctly identify the final diagnosis. Secondary endpoints: its ability to list the definitive diagnosis in the five most likely differential diagnoses and determine an adequate workup.

Results

The AI could not identify the definitive diagnosis in 2 out of the 10 cases (20% inaccuracy). Among the 8 cases correctly identified by the AI, 5 (63%) had the definitive diagnosis as the top differential diagnosis list. Regarding the suggested diagnostic tests and exams, requests for exams that would not aid in the patient's final diagnosis were made in 2 cases, representing 40% of the patients whose final diagnosis was not correctly identified by the AI. Moreover, the AI could not suggest adequate treatment for 7 cases (70%). Among them, the AI suggested inappropriate management for 2 cases, and the remaining 5 received incomplete or non-specific advice, such as chemotherapy, without specifying the best regimen.

Conclusions

Our study demonstrated GPT-4's potential as an academic support tool, although it cannot correctly identify the final diagnosis in 20% of the cases. There is also a limitation regarding the management suggested by AI. In cases where the main diagnostic hypothesis was incorrectly identified or not listed as the top differential diagnosis, the AI requested unnecessary additional diagnostic tests for 40% of the patients. Future research should focus on evaluating the performance of GPT-4 using a more extensive and diverse sample, incorporating prospective assessments, and investigating its ability to optimize diagnostic and therapeutic procedures to optimize healthcare utilization.

Clinical trial identification

Editorial acknowledgement

Legal entity responsible for the study

The authors.

Funding

Has not received any funding.

Disclosure

All authors have declared no conflicts of interest.

This site uses cookies. Some of these cookies are essential, while others help us improve your experience by providing insights into how the site is being used.

For more detailed information on the cookies we use, please check our Privacy Policy.

Customise settings
  • Necessary cookies enable core functionality. The website cannot function properly without these cookies, and you can only disable them by changing your browser preferences.