2MO - Multimodal data fusion enhanced precision neoadjuvant chemotherapy in breast cancer with a multi-task transformer-CNN-mixed learning

Background

In medical practice, clinicians merge diverse information sources. While AI has the potential to aid healthcare professionals, its current ability to smoothly integrate various algorithms and diverse multimodal data is limited, posing a constraint on its practical utilization in clinical settings. The objective of this study is to leverage AI techniques that combine histopathology and clinical data to enhance the precision of neoadjuvant chemotherapy (NAC) in breast cancer management.

Methods

We retrospectively recruited 756 patients in the training cohort from the campus 1 of Sun Yat-sen Memorial Hospital (SYSMH), 560 in the validation cohort from the campus 2 of SYSMH. Additionally, 227 patients were included in the prospective test cohort for a blinded prospective validation. We developed a AI-pathology model comprising both CNN-based and Transformer-based feature extraction channels. Building upon the AI-pathology model, we devised an AI-multimodal model that fused pathological and clinical information. The study's endpoints were pathological complete response (pCR) and disease-free survival (DFS).

Results

A total of 1,598 patients were enrolled in this study. The AI-pathology model demonstrated favorable accuracy for pCR prediction in the training cohort (AUC 0.999), as well as in the validation and the prospective test cohorts (0.995, 0.981). Notably, the AI-multimodal model (AUC 0.999, 0.994 in the validation and prospective test cohorts) surpassed the AI-pathology model. Besides, Kaplan-Meier analysis showed that patients predicted by the AI-multimodal model to achieve a pCR had a favourable DFS compared with non-pcr patients (p<0.05). The combined visualization heatmap and single-cell analysis provided insights into decision-making processes, linking model predictions with the tumor microenvironment, particularly the infiltration and functional status of T cells and B cells.

Conclusions

The AI-multimodal model, integrating both pathological and clinical information, effectively predicted pCR and DFS in the context of NAC. Its high accuracy and robustness present a novel tool for guiding personalized breast cancer management based on pre-treatment pathological images.

Clinical trial identification

Editorial acknowledgement

Legal entity responsible for the study

The authors.

Funding

This study was supported by grants 2023YFE0204000 from National Key R&D Program of China, grants 82273204 and 81972471 from the National Natural Science Foundation of China, grant 2023A1515012412 and 2023A1515011214 GuangDong Basic and Applied Basic Research Foundation, grant 2023A03J0722, 202206010078 and 202201020574 from the Guangzhou Science and Technology Project, grant 2018007 from the Sun Yat-Sen University Clinical Research 5010 Program, grant SYS-C-201801 from the Sun Yat-Sen Clinical Research Cultivating Program, grant A2020558 from the Guangdong Medical Science and Technology Program, grant 7670020025 from Tencent Charity Foundation, grant YXQH202209 from the Scientific Research Launch Project of Sun Yat-sen Memorial Hospital.

Disclosure

All authors have declared no conflicts of interest.