144P - Artificial intelligence supported treatment decisions for precision oncology

Background

Clinical interpretation of complex biomarkers for personalized treatment decisions requires extensive manual investigations of literature and databases. Recent progress in artificial intelligence (AI) and conversational large language models (LLM) suggests that these might be useful for assisting with automated screening and integration of large biomedical datasets. To better define their role in this application, we investigated several recent LLM as a support tool for precision oncology.

Methods

We generated ten fictional cases of patients with advanced cancers and genetic alterations. Each of the cases was submitted to four LLM (ChatGPT, Galactica, Perplexity, BioMedLM) and one physician as expert human annotator to identify personalized treatment options (TO). TO were blinded and presented to a molecular tumor board (MTB) if they were a) identified by a human, b) identified by more than one LLM or c) identified by a LLM and associated with clinical evidence. MTB members were asked to rate the likelihood of TO to come from an AI on a scale from 0 to 10 (0 extremely unlikely, 10 extremely likely) and to rate their usefulness.

Results

A median number of 4 and 3, 7.5, 11.5, and 13 TO per patient were identified by the human expert and four LLM, respectively. When considering the expert as gold standard, the four LLM reached median F1 scores of 0.04, 0.17, 0.14, and 0.19 over all patients combined. Combined TO by LLM reached a precision of 0.29 and a recall of 0.29 for a F1 score of 0.29 over all patients. When rated for recognizability as AI-generated, LLM-generated TO achieved a median of 8 (range 1 to 10) in contrast to 2 points (range 0 to 6) for manually annotated cases. At least one LLM-generated TO per patient was generally considered useful by MTB members. Two unique useful TO (including one unique treatment strategy) were identified only by a LLM.

Conclusions

Treatment recommendations of LLM in PO cases do not yet reach the quality and credibility of human experts. However, they can already generate useful ideas. Considering the speed of technological progress, LLM could increasingly assist with the screening and selection of relevant biomedical literature to support evidence-based, personalized treatment decisions.

Clinical trial identification

Editorial acknowledgement

Legal entity responsible for the study

The authors.

Funding

Has not received any funding.

Disclosure

D.T. Rieke: Financial Interests, Personal, Advisory Board: Bayer, Alacris Theranostics; Financial Interests, Personal, Invited Speaker: Roche, BMS, Lilly; Non-Financial Interests, Principal Investigator: Bayer. M. Schmidt: Financial Interests, Personal, Advisory Role: Mika-Health. U. Keilholz: Financial Interests, Personal, Other: Amgen, AstraZeneca, BMS, Boehringer Ingelheim, Glycotope, Innate, Lilly, MedImmune, Merck Serono, MSD, Novartis, Pfizer, Roche, Sirtex. All other authors have declared no conflicts of interest.