Oops, you're using an old version of your browser so some of the features on this page may not be displaying properly.

MINIMAL Requirements: Google Chrome 24+Mozilla Firefox 20+Internet Explorer 11Opera 15–18Apple Safari 7SeaMonkey 2.15-2.23

Poster Display session 3

4787 - A Blinded Comparison of Patient Treatments to Therapeutic Options Presented by an Artificial Intelligence-based Clinical Decision-support system


30 Sep 2019


Poster Display session 3


Translational Research

Tumour Site


Suthida Suwanvecho


Annals of Oncology (2019) 30 (suppl_5): v574-v584. 10.1093/annonc/mdz257


S. Suwanvecho1, H. Suwanrusme1, T. Jirakulaporn1, N. Taechakraichana1, P. Lungchukiet1, N. Thanakarn1, W. Decha1, W. Boonpakdee1, A. Preininger2, I. Dankwa-Mullan3, M. Solomon2, S. Wang2, G. Jackson2, V. Patel4, E.H. Shortliffe4, N. Kiatikajornthada1

Author affiliations

  • 1 Department Of Oncology, Bunrungrad International Hospital, 10110 - Bangkok/TH
  • 2 Watson Health, IBM Corporation, 02142 - Cambridge/US
  • 3 Ibm Watson Health, IBM Corporation, 20814 - Bethesda/US
  • 4 Dept. Of Biomedical Informatics, Columbia University, 10025 - New York/US


Login to get immediate access to this content.

If you do not have an ESMO account, please create one for free.

Abstract 4787


Comparison of options from clinical decision-support (CDS) systems and decisions made in practice may be biased towards the treating institution. In this retrospective study, bias was minimized by blinding evaluators to the source of treatment recommendations, either Watson for Oncology® (WFO®) or treatments patients received at Bumrungrad International Hospital (BIH), a user of WFO®.


Treatments given were compared to therapeutic options provided by WFO®. Treatments that were identical to WFO® “recommended” (green, acceptable) were not evaluated further. Paired treatments were evaluated independently in a blinded fashion by each oncologist before consensus ranking of each pair as either acceptable, acceptable alternatives, or unacceptable treatment. The consensus for each treatment was compared to WFO®, with WFO® “for consideration” (yellow, acceptable alternative), and “not recommended” (red, unacceptable). Chi-squared tests analyzed the association between risk factors and discordant recommendations.


Of 228 treatments given to patients with lung, colon, breast and rectal cancers, 174 were identical to WFO® acceptable (green) and not evaluated further; 54 non-identical pairs were evaluated (Table). Overall, 88.6% of decisions were either the same or viewed as equally acceptable by oncologists; oncologists preferred 3.9% of BIH treatments and 4.4% of WFO treatments. In cases where reasons for discordance were provided, 70% were due to BIH oncologist preference, 20% to patient preference and 10% to WFO treatment availability. We found no association between discordant recommendations and patient age or stage of cancer.Table: 1435P

TreatmentsN (%) 228 Total
Treatments are identical174 (76.3%)
Oncologists’ Evaluations
Acceptable alternatives28 (12.3%)
BIH Preferred9 (3.9%)
WFO Preferred10 (4.4%)
Both WFO and BIH-Rx unacceptable7 (3.1%)


This blinded study suggests WFO®’s therapeutic options are at as least as good as (or are an acceptable alternative to) treatments in practice. Blinding evaluators to source of treatment may minimize bias in comparisons of CDS systems and decisions made in practice.

Clinical trial identification

Editorial acknowledgement

Legal entity responsible for the study

Bumrungrad International Hospital.


Bumrungrad International Hospital.


All authors have declared no conflicts of interest.

This site uses cookies. Some of these cookies are essential, while others help us improve your experience by providing insights into how the site is being used.

For more detailed information on the cookies we use, please check our Privacy Policy.

Customise settings
  • Necessary cookies enable core functionality. The website cannot function properly without these cookies, and you can only disable them by changing your browser preferences.