ESMO Congress 2023 | OncologyPRO

Abstract 2162P

Background

There is limited data on the quality of cancer information provided by ChatGPT and other artificial intelligence systems. In this study, we aimed to compare the accuracy of information about cancer pain provided by chatbots (chatGPT, perplexity, and chatsonic) based on the questions and answers contained in the the European Medical Oncology Association (ESMO) Patient Guide about cancer pain.

Methods

Twenty questions were selected from the questions available in the ESMO Patient Guide Cancer Pain. Medical oncologists with more than 10 years of experience compared responses from chatbots (ChatGPT, Perplexity, and Chatsonic) with the ESMO patient guide. The primary evaluation criteria for the quality of the responses were accuracy, patient readability, and stability of response. The accuracy of responses was evaluated using a three-point scale: 1 for accuracy, 2 for a mixture of accurate and incorrect or outdated data, and 3 for wholly inaccurate. The Flesch-Kincaid readability (FKr) grade was used to measure readability. Stability of responses was evaluated whether the model’s accuracy is consistent across different answers to the same question.

Results

Chatbots were more difficult to read than the ESMO patient guideline (FKr= 9.6 vs. 12.8, p= 0.072). Among the chatbots, perplexity had the easiest readability (FKr= 11.2). In the accuracy evaluation, the percentage of overall agreement for accuracy was 100% for ESMO answers and 96% for ChatGPT outputs for questions (k= 0.03, standard error= 0.08). Among the chatbots, the most accurate information was obtained with chatGPT.

Table: 2162P

Comparison of ESMO and chatbots in terms of readibility and accuracy

	ESMO	Chatbots
		ChatGPT	Perplexity	Chatsonic	p-value
Readibility (FKr grade)	9.6 Easily understood	13.4 Difficult to read	11.2 Fairly difficult to read	13.9 Difficult to read	0.072
Accuracy	%100	%96	%86	% 90	0.037