Oops, you're using an old version of your browser so some of the features on this page may not be displaying properly.

MINIMAL Requirements: Google Chrome 24+Mozilla Firefox 20+Internet Explorer 11Opera 15–18Apple Safari 7SeaMonkey 2.15-2.23

Cocktail & Poster Display session

30P - Deep learning approach for discovering new predictive histological features of immunotherapy response


06 Mar 2023


Cocktail & Poster Display session


Saima Ben Hadj


Annals of Oncology (2023) 8 (1suppl_2): 100903-100903. 10.1016/esmoop/esmoop100903


H. Heuberger1, S. Ben Hadj1, R. Fick1, A. Moshayedi1, E. Rexhepaj2

Author affiliations

  • 1 Ai & Computer Vision, Tribun Health, 75015 - Paris/FR
  • 2 Data Science, Bioimaging And Computational Pathology, Sanofi - France, 75014 - Paris/FR


This content is available to ESMO members and event participants.

Abstract 30P


Immune checkpoint inhibitors treatments have proven their efficiency against Non-Small Cell Lung Cancer (NSCLC). However a large number of patients (around 45%) don’t respond to immunotherapies and despite the existing biomarkers (PDL1, CD8) and patient selection practices, there is not currently any relevant predictive biomarker of this response. The recent advances in Deep learning image analysis - namely Multiple Instance Learning - allow to train a model to predict survival for patients receiving immunotherapy and to confront visual results to known biomarkers.


2220 H&E histology images were gathered from 456 NSCLC patients. The 36-month follow-up with vital status is retrospectively documented (68.5% alive, and 31.5% deceased). The slides are first tiled into patches and histology phenotypes are extracted using K-means on VGG-16 feature extractor. The phenotypes are then fed into siamese networks and aggregated into a single patient representation through an Attention layer predicting the vital status (alive / deceased). The Attention map is finally used to extract the most important phenotypes driving the model decision of immunotherapy response. The result of the attention maps is compared with the CD8 and PDL1 biomarker stainings.


We find that a relevant number of histologically phenotypes used for the analysis is 4 and that, based on tiles coming from each of these four phenotypes the survival prediction can be performed with an AUC of 0.75. Qualitative analysis shows that the model decision for predicting response to immunotherapy from H&E stained slide not only rely on PDL1 biomarker but also how immune cells infiltrate the tumoral PDL1 area.


In this study we use a new deep learning approach enabling us to discover new histological phenotypes that predict the response to immunotherapy in NSCLC. Our first results are encouraging and demonstrate the AI capability for discovering new discriminative phenotypes. Our future work will be focused on inspecting more granular phenotypes.

Clinical trial identification

Editorial acknowledgement

Legal entity responsible for the study

The authors.


Tribun Health.


All authors have declared no conflicts of interest.

This site uses cookies. Some of these cookies are essential, while others help us improve your experience by providing insights into how the site is being used.

For more detailed information on the cookies we use, please check our Privacy Policy.

Customise settings
  • Necessary cookies enable core functionality. The website cannot function properly without these cookies, and you can only disable them by changing your browser preferences.