Regression-to-the-Mean: Insights for Drug Developers from the Sports World

Understanding Regression-to-the-Mean

The concept of regression-to-the-mean, often misunderstood, is crucial for both sports enthusiasts and drug developers. This phenomenon explains why extreme performances or conditions often return to a more typical state over time. In sports, for example, a player early in a season performing extraordinarily high or low rarely maintains that level throughout a season. While this outcome may be familiar, why it happens may be misunderstood. Frequently in sports reasons are attached to why a player does not continue the extreme performance (pressure, media scrutiny, etc.). The belief in these external myths, rather than an understanding of regression-to-the-mean, creates poor decision making in sports. The same scenario and logic apply to clinical trials, where equally misunderstood attribution of the why creates poor decision making and has massive negative effects in the drug development world.

Regression-to-the-mean can be observed in various sports scenarios. In baseball, for instance, players who start a season with exceptional stats often see their performance normalize as the season progresses. This was famously seen with Rod Carew’s and George Foster's 1977 baseball season, Foster projected to break the single season home run record at mid-season, and Rod Carew on a pace to hit the magical 0.400 batting average, finished below mid-season projections due to this statistical phenomenon. Similarly, in golf, a player's performance can vary significantly from one day to the next, purely due to statistical fluctuations rather than actual changes in skill. Understanding the performance of a player in the context of the population of players is critical to projecting the future performance. By modeling the population of players as well as the random outcomes of the player creates a recognition that players in the tails of the distribution are rare, by definition – it is more likely random outcomes that place the player the tail then in truth the player’s true skill is in the tail of the population of players. Hence any estimation of the true skill of the player will be “shrunk” to the middle of the distribution of players. Recognizing these patterns and adjusting for them appropriately greatly improves predictions, estimation, and prevents the creation of mythical attribution.

Parallel Lessons for Clinical Trials

In drug development, regression-to-the-mean is equally prevalent. Clinical trial participants often join studies at a low point in their health journey. This results in a natural improvement over the course of the trial, which can occur without any intervention. This improvement, while real, is often misattributed to the "placebo effect," when it is frequently a manifestation of regression-to-the-mean. In trial design and data analysis, it becomes critical to distinguish genuine therapeutic outcomes from these natural statistical shifts. Think of the number of clinical trials where the investigators are shocked by the size of the placebo effect. When the trial design – inclusion and exclusion criteria, focuses more and more on the tail outcomes at baseline in the trial, this will create a regression-to-the-mean for these patients – without any intervention, sham, or placebo. By believing the effect is one of a “placebo” leads investigators to make trial decisions to address potential placebo effects, when the reason for the improvement may largely be regression-to-the-mean and not a psychological placebo effect.

Drug developers must apply this understanding to avoid attributing real-world decisions to this statistical phenomenon. Take an example a single-arm trial where all patients receive the experimental intervention and there is strong improvement in the patients – is it the intervention or is this regression-to-the-mean? This understanding ensures that decision-making is grounded in a realistic expectation of outcomes over time, thus preventing costly missteps driven by misinterpreted data.

Hierarchies in Sports and Trials

The hierarchical nature of data – individual outcomes and the broader population of players, patients, subgroups, endpoints, etc. – is a key consideration when predicting future performances, whether for athletes or medical treatments. In sports, the variability between and within players influences projections. Similarly, hierarchical models in clinical trials account for variability among patient populations and responses.

In the 2017 US Open, golfers who performed poorly on day one typically improved on day two, while those who initially excelled saw declines—an illustration of regression-to-the-mean. In clinical settings, understanding these statistical principles aids in differentiating between genuine effects of a treatment and natural variability. Think about a trial with 5 subpopulations of patients – and the results in one of the subsets looks very promising. If one estimates the effect in the best performing subset only from the data in that subset they are at huge risk when they make development decisions assuming that effect is real. There are very powerful approaches to creating estimates of one unit from the results across all units that account for the regression-to-the-mean effect. Bayesian hierarchical models or frequentists random effects models allow researchers to integrate variability across the hierarchical structure, providing a more accurate estimate of treatment efficacy. The challenge is in understanding and recognizing the need for these models.

For drug development, hierarchical models can be particularly useful. They allow for between-subject differences and improve prediction accuracy in clinical trials. These models consider individual patient variability as well as overall population variability, leading to a more nuanced understanding of how a treatment might perform across diverse groups. This approach enhances decision-making, leading to more effective trial designs and patient outcomes.

Implications for Decision-Making

Understanding regression-to-the-mean is crucial not only for evaluating trial results but also for strategic decisions based on these results. Misinterpretation of isolated outcomes can result in significant downstream effects, including poor go/no-go decisions, poorly designed trials, financial loss, and opportunity costs. Mis-estimating a treatment's efficacy based on isolated results leads to inflated projections and misguided investments.

As an example, in oncology, basket trials often show considerable variability in outcomes across patient subgroups. Recognizing regression-to-the-mean provides critical context in assessing which results hold promise and what are the best estimates in the subgroups. There are many other ways this effect is manifested in drug development – the results across 10 different endpoints – how should one estimate the effect in the best or worst results across those endpoints? What if 5 different doses are used in a phase 2 trial – should one estimate the effect of one dose when ignoring the result across the other doses? When a sponsor runs 20 phase 2 trials – should they estimate the effect on the best of those trials without putting it in context of all 20 trials?

Real-World Applications and Industry Perception

The pharmaceutical industry's success relies heavily on understanding regression-to-the-mean. When preparing for regulatory submissions or deciding on continuing, modifying, or halting development, companies must factor in this statistical reasoning. It’s not natural for scientists to incorporate the results from group A to better estimate group B, but it’s critical to good decision making.

The sports analogy serves well in communicating the subtleties of these statistical phenomenon. Ideas that seem common sense in sports, seem initially awkward to drug developers. Presenting complex clinical data through familiar paradigms aids in broader comprehension among executives, stakeholders, and regulators.

Focus on Genuine Advancements

Ultimately, regression-to-the-mean underscores the importance of statistical reasoning in interpreting data. For drug developers, leveraging these insights can refine clinical trial designs and improve decision-making. By learning from parallels in sports, they can better navigate the complexities of interpreting clinical trial results. The regression-to-the-mean phenomenon may the single most important statistical concept for drug developers.