Can Borrowing Historical Data Ever Increase Power?
By Kert Viele, Ph.D.
With the recently released FDA Bayesian guidance and the upcoming EMA reflection paper, Bayesian borrowing is one of the most active regulatory topics in the world. While there is general agreement on the potential for external data to improve inferences, borrowing also presents risks, and much of the current regulatory debate covers how to assess that balance in practical clinical trials. Do we need a new paradigm, or can we fit borrowing into existing regulatory structures? In this blog we discuss potential ways borrowing can be consistent with existing frameworks with modest adjustments. Without throwing away type 1 error control, we discuss that a desire to control type 1 error may result in a preference for borrowing, rather than an aversion.
Originally motivated by a series of papers indicating that borrowing cannot simultaneously control type 1 error and increase power, and some current events where we have been asked to compare borrowing to “just raising the alpha level”, this blog accepts type 1 error control as valuable (it’s bad when ineffective therapies are approved), but focuses on precisely how that control is achieved. As with many of the topics we discuss on our blogs and podcasts, this is not an argument about math, but about the way we formulate solutions and the implied values that we attach to those solutions. At heart, we question solutions where the worst thing data can do is exist.
For simplicity, we will focus entirely on situations where we have a two arm clinical trial (control and treatment), and we are borrowing information on the control arm only. This is to limit the scope, and I think presents qualitatively different issues than borrowing treatment effects (borrowing differences between arms) across trials.
An analogy, even if it doesn’t look like one yet…
Suppose we are considering options to treat a high risk surgical population. For simplicity our outcome is either survival or death, and of course we want to minimize the death rate. We can identify the population as a whole, but the population consists of two subtypes we can’t identify in advance. We have five possible options for our surgery, with known death rates in each subtype.
| Surgery | Death rate subtype 1 | Death rate subtype 2 |
|---|---|---|
| Option A | 2.5% | 2.5% |
| Option B | 1.0% | 2.5% |
| Option C | 1.0% | 2.6% |
| Option D | 0.5% | 4.0% |
| Option E | 4.0% | 4.0% |
What should we pick, for a patient with an unknown subtype? As long as all subtypes are possible (none have 0% probability), Option B dominates Option A (lower in some, equal in all the others). Similarly, Option B dominates Option C, and all options dominate Option E. Thus B>A and B>C and everything is better than E, but all the other comparisons are harder.
Do we prefer C or A? This choice requires us to consider which subtypes may be more likely, and by how much. We may not like doing this, but it is unavoidable. Unless I think subtype 2 is much more likely, I’m going to pick C, taking its 1.5% improvement in subtype 1 and facing its 0.1% detriment in subtype 2. I can’t get out of this by saying “oh if I don’t know the likelihoods of each subtype, then I should pick A”. There are lots of situations where C is better, and if I am in one of those situations, A will cause more deaths. The tradeoffs involving option D are starker across subtypes. But again, we have pick something, and the right answer depends on the unknown subtype frequencies.
One could simply value surgeries based entirely on the worst case scenario (the worst death rate across subtypes). In this case the surgeries are cleanly ordered A=B>C>D=E. Is this the ordering we would pick? It feels odd to say A=B (or D=E) despite the dominance relationship. The lower death rate for B in subtype 1 counts for nothing in this utility. We also dismiss C, even with its decently lower death rate in subtype 1, because of the slight increase in death rate in subtype 2. There is no weighing the likelihood of the subtypes. If one subtype has a worst case scenario of 2.5%, it doesn’t matter if the other subtype has a 2.5% mortality rate or a 0.1% mortality rate, they are viewed as equal. If we indeed want to lower death rates and are faced with these choices, focusing exclusively on the worst case seems inconsistent with that goal.
Suppose I knew the proportions of each subtype at a different hospital that had some similarities to ours. We can’t guarantee the proportions are equal, but surely it is informative. Suppose that information said subtype 1 is more likely. It would seem odd to choose A in that scenario, when B is available (dominates A), or option D with its even lower mortality rate in subtype 1.
Clearly, I’ve picked 2.5% on purpose, corresponding to standard type 1 error rates. I’m going to view a false positive (a type 1 error) as something to avoid, just like the death rate in this analogy. There is nothing in this analogy that corresponds to the power of a test. My point is that even if I’m solely focused on avoiding type 1 errors (deaths), a minimax strategy (focus only on the worst case) does not necessarily, and may not even be likely to, minimize type 1 errors.
Going forward, we will discuss that most standard frequentist tests look like A or E, most Bayesian borrowing tests look like C or D, while B typically cannot be achieved on type 1 error without sacrificing power. As such, picking a design is indeed hard, but as in this analogy it doesn’t seem obvious that A or E is the right answer. In fact there may be situations where A or E look quite unappealing.
What does “type 1 error control” control?
Let’s start with what I hope is some common ground. I think approving a therapy that doesn’t work at all (a null) is a really bad thing that should happen quite rarely. It’s bad at the patient level (exposure to side effects, costs) and bad at the society level (misplaced research into me too therapies, difficulty in selecting control arms in future trials, and general degradation of trust in clinical care). I’m fully on board with minimizing this probability.
I would argue the probability we want to minimize is:
Pr(trial success AND null is true)
as opposed to the conditional probability of trial success, given the null is true (I’m about to show they are quite related, so this nuance may not matter when borrowing a control arm only, I’m bringing it up here since I think it does matter when borrowing on both arms). The penalties above only occur when both of these things happen together. I don’t outlaw a drug just because a patient has an allergy, even though the conditional probability of a bad outcome is high given the patient has an allergy. Instead, I try to avoid giving the drug to people with the allergy. It’s the dual event of the patient being given the drug and the allergy that we avoid. Similarly here we want to avoid these two outcomes (trial success and the null being true) occurring together.
This probability is difficult to compute in a frequentist framework, for a couple of reasons. We might naturally write (the “|” symbol is the conditional probability, e.g. the probability of trial success given the null is true)
Pr(trial success AND null is true) = Pr(trial success | null is true) Pr(null is true)
In the frequentist paradigm, we don’t put probabilities on parameters. Thus, Pr(null is true) is awkward. The null is either true or it isn’t. To the extent it has a probability in the frequentist framework, it is either 0 or 1 and we don’t know which.
While the first part,
Pr(trial success | null is true)
looks like a nice computable probability, it often isn’t. The reason is that “null is true” often isn’t a simple event. In a two arm trial, the null corresponds to the control and treatment arms being equal, but they could be equal to any common value M. They could both have a mean of M=3, or M=7, or M=0.5, or whatever (we call the range of possible values for M the “null space”). We typically can compute, in a frequentist framework,
Pr(trial success | null is true at M)
for any M, but not Pr(trial success | null is true) directly. In the Bayesian paradigm of course we would place a prior on M and have
Pr(trial success | null is true) = integral_M Pr(trial success | null is true at M) pi(M) dM
Within the frequentist paradigm, we can’t formulate that distribution over M. Given the null is true, M must have some value, but we don’t know what.
As such, if we want to compute the true value of
Pr(trial success AND null is true) = Pr(trial success | null is true) Pr(null is true)
we can’t do it directly in a frequentist framework. But we can create an upper bound. From above Pr(null is true) is 0 or 1, so it’s at most 1. Suppose we can find a value alpha such that Pr(trial success | null is true at M) <= alpha for all M. Combining these, we get
Pr(trial success AND null is true) <= alpha
and this is standard frequentist type 1 error control. As such, the statement “the type 1 error rate is the probability of trial success, given the null is true” is becoming somewhat of a pet peeve of mine. It is NOT that probability, but rather an upper bound on that probability, one which might be good or bad (preview of coming attractions, in Bayesian borrowing it’s bad).
Thus, we can see one of the reasons that a “minimax” strategy (like “judge a surgery only by the worst case”) might enter into our calculations. It’s the only way to avoid the problem of putting a distribution over M. In my surgery analogy, the two subtypes reflect two possible values of M (which of course would typically form a continuum). In what follows, we will argue that when borrowing only on control, borrowing can control the real probability Pr(trial success | null is true) but not the upper bound. We won’t talk about situations where you borrow the treatment effect (both arms), which will rely on arguments that the historical information is controlling Pr(null is true) (the past information may say this probability is small apriori). With this qualitative difference, it’s reasonable to me that regulators may find borrowing on control more palatable than borrowing on the treatment effect (though I think both can be justified in the right settings).
How standard frequentist tests are constructed, and why we get surgery A or E
Given the emphasis on the worst case upper bound, a standard mechanism for choosing tests is to specify that upper bound (often one sided 2.5%), look at all tests which have that worst case upper bound, and then pick the test with the highest power. There are lots of interesting mathematical results along this line (this is pretty standard material for first years master’s level statistics). This is an important value statement about type 1 error. We are stating that type 1 errors of 2.5% and 2.501% are very different (one exceeds the bound), but type 1 errors of 2.5% and 0.1% are the same (they both obey the bound).
Type 1 error and power are related. They both involve the probability of trial success. A test specifies what data will result in our claiming trial success (e.g. “the drug works”). Typically these look like “we will claim success if the observed treatment effect exceeds X”. When X is small as opposed to large, we claim success more, and claiming success more will increase power (the probability of claiming success when the drug works) as well as increase type 1 error (the probability of claiming success when the null is true). This occurs for every M in the null space.
If we receive no credit for reductions of type 1 error below the bound, we may as well run the type 1 error to the upper bound everywhere. We want Pr(trial success | null is true at M) = alpha everywhere to maximize power. Thus, we will end up with tests that look like surgeries A and E, with flat, or approximately flat, rates (across the surgery subtypes in that analogy, or across values of M in our tests). This is a consequence of how we measure type 1 error, not the concept of type 1 error itself.
What about those papers that say we can’t get power gains from borrowing? Why can’t we get surgery B that dominates A?
Borrowing information does not change the result above. The type 1 error and power of a test are defined by the rules connecting data to trial success, and borrowing does not change the potential data outcomes of our current trial. The result above still holds.
Thus, tests like B typically aren’t in the running for “best test”. While we could in theory get a test that maintains an equal upper bound and achieves localized type 1 error reductions, it will always come with decreased power, and this will always be viewed negatively by sponsors (if you do care about lowering type 1 error and aren’t a minimaxer, this reduction in type 1 error may compensate for the reduced power, but typically that isn’t proposed). At some level, the minimax strategy we introduced to control type 1 error discourages us from reducing type 1 error.
Exception – in some cases we might consider the entire data generating mechanism, including the data we are borrowing. For example, before starting trial 1, we might prospectively indicate that we will borrow trial 1 into trial 2. If we calculate the type 1 error and power of the program as a whole, we might increase power by borrowing while controlling minimax type 1 error (this requires assumptions on the relationship between the trial 1 and trial 2 parameters). This would allow us to get surgery B as an option. In practice our historical data is fixed and we only consider the data generation mechanism from trial 2 in computing power and type 1 error.
What about surgery options C and D and their relationship to Bayesian borrowing?
We indicated that surgery B (which dominated A on type 1 error) is not obtainable without sacrificing power. But surgery options C and D are available with increased power. The key features of surgery C and D are the decreased mortality rates for certain subtypes with increased mortality rates for others (this will correspond to decreased type 1 error in certain parts of the null space and increased type 1 error in others).
The intuition behind borrowing is that we hedge our bets toward the historical data. When our current trial matches history (this will be analogous to surgery subtype 1), this hedging produces inferential gains. In this case we get more precise estimates, with minimal bias and thus better mean square error, and we can get both lower type 1 error AND increased power. But this is localized, not everywhere. When our current trial does not match history, our hedging toward the historical data is a bad idea, and we get biases and increased error rates (I’m avoiding some details that we get either increased type 1 error or lower power depending on the direction of the mismatch). This is going to correspond to subtype 2 (at least the direction where we increase type 1 error, which is our focus here). If you hedge toward something, you are hedging away from something else. You can’t get pure gains everywhere.
Thus, we can get options like C and D, with lower rates in some subtypes and higher rates in others (and while there was nothing in our analogy involving power, in the borrowing setting we can get increased power where we get the type 1 error gains).
In the borrowing setting, the difference between C and D, where D presents more stark tradeoffs than C, is the degree of borrowing. If we choose to borrow modestly, we get a type 1 error picture looking like surgery C, with modestly increased power where type 1 error is improved. If we choose to borrow aggressively, we get something closer to D (or even more extreme), with large, localized savings in type 1 error and large inflation of type 1 error in the worst case (again while power is not part of the surgery analogy, the power gain/loss can be equally stark). Our surgery analogy is not rich enough, but with borrowing option D will also come with a smaller range of M where we obtain benefit. Aggressive borrowing gives great benefits in a narrow range of M, modest borrowing produces modest benefits, but in a greater range of M
What’s a trialist to do?
Just like the surgery example, where comparisons between C and D versus A were difficult because of the lack of a dominance relationship, we are going to end up with the same issue in testing. The minimax frequentist paradigm will lead us to trials like A, with constant type 1 error for all values in the null space and the best power subject to that upper bound constraint. In contrast, borrowing is going to provide choices where (1) some parts of the parameter space have both reduced type 1 error and increased power, and (2) other parts of the parameter space where we get increased type 1 error (or lower power which we haven’t discussed as much).
There is no dominance relationship. Much like in the surgery example where we have to consider which subtype is more likely, in deciding among tests which is better depends on the relative likelihood of these regions of the parameter space, which are unpleasantly unknown. If we are trying to minimize the actual value of Pr(trial success and null is true), this unavoidably depends on the unknown value of M.
One argument in favor of borrowing is that the historical data itself provides some information on what parameter values are most likely. If we can trust the historical data (typically based on having similar populations, similarity in time, etc.), then we will be in precisely the situation where Pr(trial success and null is true) is lower when we borrow than when we don’t. At the same time, we have to face the risk we are wrong. To me, we collect data because we think we will learn something from it. If we collect and assess data in good faith (and yes, some sponsors drastically overestimate the relevance of their historical data) we should be able to control the relevant probability when borrowing.
In fact, at some level the minimax strategy of type 1 error control leads to a bizarre conclusion for the statistician. What is the worst case scenario? That all the historical data were generated to maximally deceive us. Hopefully this isn’t true (if it is, we have bigger problems than type 1 error control will solve). Essentially, we view all the yet unobserved data in the future as unbiased and good, while observed data from the past must be viewed as bad. In the minimax strategy, the worst thing data can do is exist. In practice, historical data will not be a perfect match for the current trial (concurrent trials are indeed more valuable than historical ones). Weighed correctly, the borrowed data augments the current control arm. It does not replace it.
How might we view this tradeoff outside of a minimax strategy?
As stated above, I have no problem trying to minimize Pr(trial success AND null is true). I would be inclined to balance this with power, the full range of treatment effects and so forth, but all other things being equal keeping this probability low is valuable. Keeping this probability low is not the root cause of the difficulties with borrowing. Borrowing on a control arm can result in the actual value of this probability being quite low. What borrowing can’t do is control the worst case scenario. We don’t need to abandon type 1 error, but we do need an alternative to the minimax paradigm.
One option above was to consider the entire sequence of events, including the generation of the historical data. Again, this is often not done because the historical data is typically fixed at design time.
We can consider other functions of the type 1 error curve. We can look at
Pr(trial success | null is true at M)
across M and take values other than the worst case scenario. This may be justified because the worst case scenario may be a very unlikely scenario. Some methods specify a range of M (often centered on the historical data) and only look at the worst case over that range, granting more flexibility, assuming of course the global worst case is not in that range. This is similar to only looking at certain subtypes in the surgery example.
Alternatively, we might consider expected type 1 error rather than worst case type 1 error. In the surgery example, if I knew the proportions of patients in each subtype I would be inclined to average the mortality rates over the subtype frequencies to get an overall mortality rate, and use that to order tests. In the borrowing setting, many authors have proposed expected type 1 error rates (and expected power), by placing a distribution over M and computing a weighted average of Pr(trial success | null is true at M) with weights from that distribution. This is analogous to the averaged mortality rate. This is natural in a Bayesian framework where we are effectively placing a prior on M. Within the frequentist framework it is less natural, but again the fundamental problem is that the subtypes differ, and we have to do something to decide, and that something depends on what subtypes are most likely. We can look at a range of such distributions for robustness. In our surgery example, different subtype frequencies may result in our preferring option C or option D. Similarly, with borrowing, different distributions will result in our preferring more or less aggressive borrowing, including the possibility of no borrowing.
There are also proposals to adjust the minimax strategy so that instead of selecting a global upper bound for type 1 error, we specify a function for Pr(trial success | null is true at M) that varies with M. I think these are interesting, but they continue to locally have a “cliff” utility. There is still a bound at each M, and local savings on that bound aren’t valued.
What about “just increasing alpha”?
I have been in meetings before where borrowing has been considered “complex” and the suggestion has been made to just increase the alpha to “keep things simple”. Essentially the argument is “the worst case scenario for your proposed borrowing is X, let’s just agree to change the upper bound to X, but compute a frequentist test”. This essentially says we consider surgery D’s death rates to be too complex, and prefer surgery E as a replacement. Surgery E has the same maximum, but it heavily increases the mortality rate in subtype 1. With borrowing, just increasing the alpha may keep the maximal type 1 error rate equal, but it typically comes with large type 1 error increases, precisely in regions of the null space we may think are most likely. This seems very odd to me. Essentially it argues that to avoid type 1 error inflation (defined by the maximum), we should have MORE type 1 error inflation in all the region where borrowing had reduced Pr(trial success | null is true at M). It seems if you care about type 1 error, you should instead borrow.
Conclusion
I’ve tried to go through several of the standard sticking points I have encountered with borrowing, and I’ve tried to review the smallest changes to the standard minimax type 1 error paradigm that would allow borrowing. At heart, a lot of these suggestions involve doing one’s best to compute the real value of Pr(trial success and null is true), as opposed to only being concerned about an upper bound. If type 1 error truly is a concern, minimizing the real probability, while difficult, is paramount, and it requires consideration of the likely values within the null space. If we believe collecting data is generally valuable, and we act in good faith to both use all relevant data and downweight or ignore tangential data, we can borrow without breaking (in fact furthering) the regulatory paradigm of avoiding false positive conclusions. The frequentist paradigm values long run behavior of methods. In a regulatory setting, we should see data lead us in the right direction more often than the wrong direction, and thus borrowing should make our long run inferences better.