672P - Inference failure with synthetic arms: Empirical application to phase III oncology trials

Background

Open sharing of synthetic trial data could facilitate secondary analyses and synthetic control arm development.

Methods

This study included 128 experimental vs. control treatments for solid tumors from 115 phase 3 randomized clinical trials (RCTs) published in 7 high-impact journals in 2023. Original individual patient data were reconstructed from the published Kaplan-Meier curves of overall survival (OS) in the intention-to-treat population. Either both arms (fully synthetic; retaining original sample size) or only the control arm (partially synthetic; retaining original control arm size) were synthesized using statistical models (Weibull and Synthpop) and a deep generative model (SurvivalGAN). Original and synthetic RCTs were compared through the probability overlap (0: none, 1: perfect) of the 95% confidence intervals (CIs) for the hazard ratio (HR), and the true (TPR) and false positive rate (FPR) with respect to the direction of the HR and its statistical significance on the 0.05 level.

Results

The median sample size was 571 (range 82-5637) and the median HR for OS was 0.80 (range 0.39-1.14) in the original (reconstructed) RCTs. Fifty-six (43.8%) were statistically significant, all of which showing superiority of the experimental treatment. The results presented in the table indicate that the use of synthetic arms may introduce bias due to generative model misspecification (in particular for fully synthetic RCTs created by Weibull) and additional variability inherent to the data generating process (more pronounced in fully vs. partially synthetic RCTs and in deep generative vs. statistical models), leading to insufficient CI overlap for the HR and thereby impacting statistical inference. TPR ranges from 2.6% to 80.4% and 90.9% to 98.2% in fully and partially synthetic RCTs, while FPR ranges from 25.0% to 46.4% and 27.8% to 44.4%, respectively. Table: 672P

Results of the synthetic RCTs

Conclusions

Naive inference from RCTs with synthetic arms may lead to overly optimistic or even wrong conclusions.

Clinical trial identification

Editorial acknowledgement

Legal entity responsible for the study

Ghent University.

Funding

Has not received any funding.

Disclosure

All authors have declared no conflicts of interest.