2066P - Evaluation of the quality and reliability of ChatGPT and Perplexity's responses about rectal cancer

Background

This study aimed to assess the usefulness of ChatGPT and Perplexity's responses to frequently asked questions regarding the diagnosis and treatment of rectal cancer.

Methods

Both artificial intelligence systems were asked 20 questions covering disease risk factors, screening, symptoms, treatment, and treatment side effects. In accordance with the ESMO patient guides, the questions were posed in the order specified. According to ESMO patient guide recommendations, responses were graded as “appropriate,” “inappropriate,” or “insufficient.”

Results

While ChatGPT provided appropriate responses to 17 of the questions (85%), the responses to two questions (10%) were deemed inappropriate, and the response to one question (5%) was deemed insufficient. Nineteen (95%) of Perplexity's responses were deemed appropriate. It was determined that both artificial intelligence systems responded improperly to the nutrition question.

Conclusions

The study's findings indicate that ChatGPT and Perplexity provide patients with accurate information about rectal cancer. The accuracy rate of Perplexity was found to be higher than that of ChatGPT. It can be demonstrated that the benefit of Perplexity is that it displays the source for each piece of information. However, especially cancer patients require emotional and psychological assistance, and further research is required to determine the impact of these artificial intelligence systems on the patient's psychology. Moreover, given the complexity and individuality of medical care, AI chatbots should not be viewed as a replacement for medical professionals, but rather as a supplement that can help patients gain access to trustworthy information and support. Overall, this study contributes to the growing body of literature on AI chatbots and their potential use in cancer by highlighting both the advantages and disadvantages of these technologies.

Clinical trial identification

Editorial acknowledgement

Legal entity responsible for the study

The authors.

Funding

Has not received any funding.

Disclosure

All authors have declared no conflicts of interest.