67P - Multi-institutional evaluation of interrater agreement of biomarker-drug pair rankings based on the ESMO scale for clinical actionability of molecular targets (ESCAT) and sources of discordance

Background

The widespread use of next generation sequencing has contributed to the need to identify strategies on how to select the most relevant alterations for targeted therapy. The ESCAT framework was designed to provide guidance for the ranking of biomarkers based on the levels of evidence (LOE) that reflect their clinical significance and actionability based on the published literature data. The objective of this study was to determine whether the ranking of LOE for biomarker-drug pairs based on the ESCAT system is dependent on the human factor.

Methods

To evaluate the inter-rater agreement, we created a dataset of a total of 154 biomarker-drug pairs for 18 unique tumor types. We aimed to include biomarker-drug pairs that could be considered standard of care as well as less common and under investigated pairs. Fourteen precision oncology experts were invited to assign an ESCAT level of evidence for biomarker-drug pairs. Statistical analysis was carried out using Cohen’s kappa and Kolmogorov–Smirnov test.

Results

According to the results, the inter-rater agreement was low with some exceptions, and significant deviations from the consensus level of evidence were observed. For biomarker-drug associations the deviations from the consensus were observed for more than 50% of the contributors’ rankings. The most agreement between the contributors was observed for lung adenocarcinoma (p-value < 0.005), while the most disagreement was observed for esophageal cancer (p-value < 0.01) biomarker-drug pairs in our dataset.

Conclusions

This study demonstrates noteworthy discordances between the precision oncology experts and may provide the directions for future developments in modifying the ESCAT framework and the overall applicability of the results of genomic profiling into the clinical practice.

Editorial acknowledgement

Clinical trial identification

Legal entity responsible for the study

The authors.

Funding

Has not received any funding.

Disclosure

All authors have declared no conflicts of interest.