Digital Googols and the Future of Clinical Learning
What Digital Twins Mean for Modern Trials
The term “digital twin” comes from engineering, where virtual models of a physical system – machines or systems enable an in-silico copy of the physical system. The power of quality digital twins is enormous. In engineering the design, robustness, and optimization of the physical process can be done with virtually no cost and with immediate and precise effects. Imagine a complete digital version of an airplane – with the ability to optimize the design for fuel efficiency, safety, and robustness, without any cost of building thousands of versions of the airplane. Clinical trialists are familiar with the practice of building digital twins of a clinical trial design. Clinical trial simulation allows a complete digital version of the full physical trial. This allows optimization of the design – across a wide range of possible truths. This allows for the optimization of the design for efficiency, strength, and safety – before the actual physical trial has ever been conducted.
While clinical trial simulation is a powerful use of digital twin technology – this is not the type of digital twin that is creating excitement and controversy in the clinical research space. Digital twin terminology is used as a way to build a digital version of a patient in a clinical trial. The potential for this patient-level digital twin technology is huge. If one could create a digital copy of a patient that is assigned to an investigational treatment and the counterfactual of what would happen without the investigational treatment, it would enable an incredible tool to understand the effect of the treatment with much less uncertainty and no control arm.
Are we there yet? Can we build a digital prediction of a patient on a control? Yes and no. There are many conditions where the behavior on a control arm is very well understood and there are great data resources. ALS is one such disease. In part ALS is a good example because of the severity of the disease and the tremendous data sources available (the PRO-ACT data base). Even in ALS, and in most diseases, we do not have the data necessary – and when we do the uncertainty of what would happen to the patient under control is significant – typically this uncertainty is beyond the size of treatment effects and concerns about comparability and appropriateness of models. While we aren’t there – we are getting there very rapidly and the future is quite promising.
Regulatory Qualification of PROCOVA
The European Medicines Agency has qualified one company’s use of the digital twin technology, Unlearn’s branded methodology, PROCOVA. PROCOVA employs, by their website, deep neural networks trained on historical disease data to predict an endpoint—like cognitive decline in Alzheimer’s disease—at a set future timepoints. For what is this qualified? The digital twin model based prediction is used as a covariate adjustment in a randomized trial analysis. The predicted value is the baseline covariate – which if the prediction is a good prediction, its use would reduce the variability of the change from this baseline estimate. The theory is this will lower variability in measured treatment effects, enabling smaller, faster trials. The approach is quite reasonable, but what’s the big deal?
Berry’s evaluation is direct: the method is an extension—not a reinvention—of classical covariate adjustment, a statistical standard familiar to all trialists. Existing models already fit baseline variables (e.g., sex, baseline cognition, age, etc.) as covariates in a primary analysis. Therefore, the PROCOVA is only better than what we always do if the predicted value from the confidential modeling is better than the prediction that comes from the baseline covariates. If a statistician had the same data as the PROVOVA model they could optimize the covariates used – in a way that is equivalent to the use of the PROCOVA. PROCOVA merely substitutes an opaque, proprietary neural net for transparent, interpretable adjustment models. There is, to date, no convincing evidence that PROCOVA’s “AI-powered” covariate delivers better adjustment than a simple regression model applied to the same baseline data. As Dr. Scott Berry puts it, “I highly doubt in most scenarios that…this is actually better.”
The skepticism is particularly sharp where model details are withheld. Proprietary black-box approaches applied to confidential data subverts reproducibility; sponsors and regulators cannot interrogate, replicate, or improve their operation. Scientific and regulatory progress depends on a clear record of model assumptions, accessible code, and independent verifiability.
Digital Googols
The use of virtual predictions for a patient in a trial is not new – despite the hype in the use of “digital twins” and AI-based terminology. Disease progression models used for modeling potential outcomes has been used and is powerful – when the data are robust and appropriate. When this technology is used one must characterize the variability of the potential outcome. A single “twin” of a patient on a treatment is wholly inadequate. Can we create many virtual outcomes for a single patient – say a googol (10^100) of them? Mathematically we understand this as a probability distribution of the potential outcome. Having this probability distribution (or many simulated trajectories) per patient would be very powerful for understanding the effect of a treatment.
Crucially, scientific modeling cannot rely on a single average prediction per patient—the “Alexa, what would have happened?” approach Berry lampoons in the podcast. Only through generating full distributions can uncertainties be captured and appropriate probability statements made.
Using therapeutic areas like ALS (where the PRO-ACT database is available), statisticians employ transparent, open statistical models to predict patient trajectories based on historical controls and current baselines. Propensity score modeling and virtual control approaches are also powerful possibilities. At minimum these methods and data used need to be disclosed and available for scrutiny.
When these model-based estimates, based on incredibly rich data, at the patient-level, are available, there is a potential for enormous impact to clinical research. Yet those same developments present their own set of barriers: data remain siloed, access is inconsistent, and model complexity rises alongside the challenge of interpretation and validation.
The promise of digital googols, like all model-based advances, is real and we will get there – there is no alternative. We need and will get the appropriate data. Even 10 years ago there were very little openly available data sets – now we have many wonderful examples – and the quality, richness, and breadth of these data sets are rapidly improving.