Abstract 2162P
Background
There is limited data on the quality of cancer information provided by ChatGPT and other artificial intelligence systems. In this study, we aimed to compare the accuracy of information about cancer pain provided by chatbots (chatGPT, perplexity, and chatsonic) based on the questions and answers contained in the the European Medical Oncology Association (ESMO) Patient Guide about cancer pain.
Methods
Twenty questions were selected from the questions available in the ESMO Patient Guide Cancer Pain. Medical oncologists with more than 10 years of experience compared responses from chatbots (ChatGPT, Perplexity, and Chatsonic) with the ESMO patient guide. The primary evaluation criteria for the quality of the responses were accuracy, patient readability, and stability of response. The accuracy of responses was evaluated using a three-point scale: 1 for accuracy, 2 for a mixture of accurate and incorrect or outdated data, and 3 for wholly inaccurate. The Flesch-Kincaid readability (FKr) grade was used to measure readability. Stability of responses was evaluated whether the model’s accuracy is consistent across different answers to the same question.
Results
Chatbots were more difficult to read than the ESMO patient guideline (FKr= 9.6 vs. 12.8, p= 0.072). Among the chatbots, perplexity had the easiest readability (FKr= 11.2). In the accuracy evaluation, the percentage of overall agreement for accuracy was 100% for ESMO answers and 96% for ChatGPT outputs for questions (k= 0.03, standard error= 0.08). Among the chatbots, the most accurate information was obtained with chatGPT.
Table: 2162P
Comparison of ESMO and chatbots in terms of readibility and accuracy
| ESMO | Chatbots | ||||
| ChatGPT | Perplexity | Chatsonic | p-value | ||
| Readibility (FKr grade) | 9.6 Easily understood | 13.4 Difficult to read | 11.2 Fairly difficult to read | 13.9 Difficult to read | 0.072 |
| Accuracy | %100 | %96 | %86 | % 90 | 0.037 |
Conclusions
The results suggest that ChatGPT provides more accurate information about cancer pain compared with other chatbots.
Clinical trial identification
Editorial acknowledgement
Legal entity responsible for the study
Gazi University Ethic Commitee.
Funding
Has not received any funding.
Disclosure
All authors have declared no conflicts of interest.
Resources from the same session
2247P - Expression of PD-L1 on circulating epithelial tumor cells (CETCs) in men with primary non-metastatic prostate cancer
Presenter: Dorothea Schott
Session: Poster session 07
2248P - Capecitabine-based concomitant chemoradiation followed by durvalumab as a neoadjuvant strategy in locally advanced rectal cancer (PANDORA trial): Molecular profiling and translational results
Presenter: Stefano Tamberi
Session: Poster session 07
2249P - Mitigation of tumor microenvironment-mediated immunosuppression using a PD1-41BB switch protein with optimal affinity tcrs for first-in-class, 3rd generation TCR-T therapies
Presenter: Kirsty Crame
Session: Poster session 07
2250P - Exploring tumor microenvironment in lung cancer through patient-derived 3D models
Presenter: NORA JUCIUTE
Session: Poster session 07
2251P - The genomic landscape of radiotherapy-induced breast angiosarcoma: An alleanza contro il cancro (ACC) initiative for an unmet need in rare tumors
Presenter: Lorenzo Ferrando
Session: Poster session 07