Analyze continuous data!

Jim Garrett

  1. Introduction
  2. Why binning is so seductive
  3. Statistical efficiency
  4. Representing Nature

Introduction

Over and over I’ve found my data analysis strategies to be contrary to those of many of my peers. One area of differing inclination is that, when given continuous data, I try very hard to analyze it as continuous, rather than binning it. My preference is, according to standard principles, the theoretically preferred approach: binning is discarding information and loss of information must lose statistical efficiency. Nevertheless, this textbook advice is almost universally ignored.

I’m going to argue here that binning is a bad practice, but not for statistical efficiency reasons. Instead, I argue for trying hard to analyze continuous data on the grounds of clarity. The more we meet Nature where she is, the more clearly we can understand her and make reasonable decisions.

Note: I’ve realized that this blogging platform doesn’t allow one to include figures, unless one has access to the server, which I don’t. I’m in the process of setting up my own web site, at which point I’ll move my blog there. In the meantime, in lieu of figures, I’ll include code to produce the figures. Perhaps this will offer a little tutorial benefit. At least, I’m trying to make lemonade out of a lemon.

Here’s some setup code:

library(rms)
library(mgcv)

## A crude hand-sketched example
template <-
    data.frame(x = c(0, 4, 5, 6, 7, 8,    9,  10),
	       y = c(0, 0, 1, 3, 4, 4.25, 4.5, 4.75))

## A function to linearly interpolate
tempFun <- 
    approxfun(x = template$x,
	      y = template$y)

## Generate 
set.seed(123)
datbin <- data.frame(x = runif(300, 0, 10))

## Transform template to probability scale, ranging from 0.1 to 0.8
invLogit <- function(r) 1 - 1 / (1 + exp(r))

LogitLow <- log(0.1 / 0.9) # -2.197225
LogitHigh <- log(0.8 / 0.2) # 1.386294

datbin$p <-
    invLogit((LogitHigh - LogitLow) / tempFun(10) * tempFun(datbin$x) + LogitLow)

MedianBin <- median(datbin$x) # 4.778207

datbin$XGp <-
    factor(ifelse(datbin$x >= MedianBin, "High", "Low"),
	   levels = c("Low", "High"))

set.seed(456)
datbin$YBin <- rbinom(nrow(datbin), size = 1, prob = datbin$p)

Why binning is so seductive

As mentioned, virtually any academic statistician will argue against binning categorical data on efficiency grounds, and yet the practice is pervasive. Why? I don’t know for sure, but my guesses are as follows.

First, our customers ask for it. Very often a statistician’s customer is a medical clinical researcher, and many clinical researchers are accustomed to seeing a comparison of discrete groups. In fact, many clinical researchers aren’t even aware that any analysis other than comparing groups is even possible, if the response is not continuous. For instance, if the response variable is binomial with a hypothesized probability p that depends on independent study factors, a non-statistician researcher may imagine that a probability is a property of a group, i.e., a proportion of the whole. To evaluate a probability of an outcome given a study factor, we must define a subgroup based on the factor and then count successful outcomes per subgroup. Modelers know that logistic regression can describe the probability p as changing continuously as a function of other factors: it’s possible for every single study subject to have their own distinct probability p.

Therefore a clinical researcher asks for a subgroup definition because they prefer to think that way or they’re unaware of alternatives. Next, the pliable statistical analyst may want to please his customer by fulfilling the request as quickly and directly as possible. This raises questions about what a statistical analyst’s ideal role in collaboration is, but that’s for another day….

Second, binning can simplify analysis. Continuous data cannot be depended upon to be nicely normally-distributed as in textbook cases. There can be non-normal distributions, skewness, outliers, etc. When relating to other variables, we similarly cannot rely on relationships being linear. All of these problems simply go away when we bin. Deciding not to bin is not a simple choice, but is rather a commitment to cultivate an entirely new toolbox of analysis strategies suitable for messy cases.

Given the analyst caught between customers demanding binning on the one hand, and maybe not fully confident about their continuous-data toolbox on the other hand, perhaps it’s no surprise that most analysts simply provide the customer with what the ask for.

Still, I challenge statistical analysts to start nurturing that toolbox. If you want to know how, watch this space.

Statistical efficiency

Let’s consider the question of statistical efficiency and power with a simple hypothetical example. Suppose we are assessing a possible biomarker in a clinical trial, and the biomarker is assayed on 300 subjects. Biomarker values are uniformly distributed over a range. Suppose the clinical outcome is binary. An increased biomarker value contributes to an increased probability of clinical response, and the true relationship is per the piecewise-linear curve generated as follows:

plot(0:10,
     invLogit((LogitHigh - LogitLow) / tempFun(10) * tempFun(0:10) + LogitLow),
     type = "l",
     ylim = c(0, 1),
     main = "True probability of response",
     xlab = "Biomarker value",
     ylab = "Probability of response")

(This produces a piecewise-linear curve that is somewhat sigmoidal, starting from a probability of 0.1 (where it remains for some time), and then it increases rapidly to roughly 0.7. Then it continues increasing, but at a slower rate. Over the range of the curve (from 0 to 10), it reaches a maximum probability of 0.8.)

Even though the curve is crudely piecewise linear, it has some features that complicate real-world analysis:

We want to assess whether the biomarker is worth investigating further, and if so, the nature of the relationship. I’ll carry out three alternative strategies:

  1. Calculate the median value of the biomarker and split cases into biomarker “High” and “Low” groups accordingly. Apply Fisher’s Exact Test to test for association between binary biomarker group and binary clinical outcome.
  2. Fit a generalized linear model relating binary outcome to a smoothing spline estimate of the biomarker’s contribution. Use an approximate test against the null model of no relationship. Optionally, we can assess evidence for non-linearity. Obtain an estimate of the relating curve, with pointwise confidence intervals.
  3. Similarly, fit a logistic regression model, but use a natural spline expansion for continuous biomarker values. Obtain a likelihood-ratio test against the null of no relationship, and as with the GAM, assess the evidence for non-linearity and obtain an estimate of the curve mediating the relationship.

Should we include fitting a continuous model that assumes linearity? I think not, because we don’t know if that’s the case, and a non-linear relationship is quite plausible.

With alternative (1), Fisher’s Exact Test gives a p-value “< 2.2e-16”. Statistical efficiency is not an issue here.

(The following code evaluates Fisher’s Exact Test with the X grouping against the binary outcome.)

fisher.test(datbin$XGp, datbin$YBin)

With the GAM–alternative (2)–we also find <2e-16 as a p-value against the null hypothesis of no relationship. The following code generates an estimate of this relationship:

GamMod.init <- gam(YBin ~ s(x), family = binomial, data = datbin)

summary(GamMod.init)

plot(GamMod.init,
     trans = invLogit,
     ylim = c(0, 1),
     main = "Logit outcome vs biomarker",
     xlab = "Biomarker",
     ylab = "Logit of outcome probability")

(The figure shows a figure that tracks the true curve, but it is implausibly “wiggly”.)

This tracks the true curve but is too wiggly to be plausible. Manually forcing the smoothing parameter to be large enough so that the curve is almost monotonic, we find:

GamMod <- gam(YBin ~ s(x, sp = 0.05), family = binomial, data = datbin)

plot(GamMod,
     trans = invLogit,
     main = "Logit outcome vs biomarker",
     xlab = "Biomarker",
     ylab = "Logit of outcome probability")

(The code generates a less-wiggly curve that still tracks pretty well.)

The actual predicted probabilities are not very different between these estimates, actually, and they both give p-values of <2e-16 against the null model.

## initial model
summary(GamMod.init)

## Add a linear component to the initial model so that linear and non-linear 
## components can be assessed separately.
GamMod.init.lin <- gam(YBin ~ x + s(x), family = binomial, data = datbin)

summary(GamMod.init.lin)

## smoother model
summary(GamMod)

(The initial and the smoother GAM models both show a p-value for the relationship betweeen X and outcome of <2e-16. Additionally, a linear term is added to the initial model in order to assess the non-linear contribution separately. This shows that p-values for the linear and the non-linear contributions are both very small.)

Applying alternative (3), unpenalized spline regression, we obtain the following curve:

## Use rms package to enable nice ANOVA
RegModBin <- lrm(YBin ~ rcs(x, parms = 5), data = datbin)


## Use base or "stock" glm to support likelihood ratio test via base 
## anova function

RegModBin.s <- glm(YBin ~ rcs(x, parms = 5), data = datbin)

## Plot
## Set of points on X axis for plotting
TmpSeq <- seq(0, 10, length = 200)

## Get predictions on logistic scale, then calculate confidence limits
## on that scale, then transform to probability scale
Preds <-
    predict(RegModBin,
	    newdata = data.frame(x = TmpSeq),
	    type = "lp", se.fit = T)
## has components "linear.predictors" "se.fit"

CIReg <- 
    data.frame(Est = invLogit(Preds$linear.predictors),
	       Low = invLogit(qnorm(0.025,
				    mean = Preds$linear.predictors,
				    sd = Preds$se.fit)),
	       High = invLogit(qnorm(0.975,
				     mean = Preds$linear.predictors,
				     sd = Preds$se.fit)))

plot(range(TmpSeq), c(0, 1), type = "n",
     main = "Outcome probability vs. biomarker",
     xlab = "Biomarker",
     ylab = "Outcome probability")
polygon(x = c(TmpSeq, rev(TmpSeq), TmpSeq[1]),
	y = c(CIReg$Low, rev(CIReg$High), CIReg$Low[1]),
	col = "thistle", border = NA)
lines(TmpSeq, CIReg$Est)
rug(datbin$x[datbin$YBin == 0])
rug(datbin$x[datbin$YBin == 1], side = 3)

In this plot the Y-axis is on the probability scale rather than the logit scale. This is substantially equivalent to either GAM model. Here we can give an informative ANOVA breakdown:

## Generate the ANOVA table
anova(RegModBin)

Factor     Chi-Square d.f. P     
x          69.85      4    <.0001
 Nonlinear  9.89      3    0.0195
TOTAL      69.85      4    <.0001

## Likelihood ratio test using base R
anova(RegModBin.s, glm(YBin ~ 1, data = datbin))

This indicates that there is overwhelming evidence that the biomarker influences the outcome, and furthermore there is strong evidence of a nonlinear component, i.e., a departure from linearity on the logit scale. While this representation doesn’t show exactly how small the p-value is, a standard likelihood-ratio test yields <2e-16, just as the other methods.

In summary, all three approaches indicate strong evidence that the biomarker influences the clinical response. This does not support the idea that the continuous approach is more powerful. The median split represents the biomarker with 1 degree of freedom, while the continuous approaches use roughly 4 degrees of freedom. They yield more information, but “cost” more. It’s a fair trade, but it’s not clear that one always has more power than the other.

Rather, the reason that I recommend a continuous approach is that, in one step, we (1) assess evidence for a non-null relationship and (2) gain a reasonable estimate of that relationship. Further, we do the latter without carrying out substantial optimization or multiple looks at the response, which compromises statistical reliability.

Now let’s think a little more about real life.

Representing Nature

Here’s a true story illustrating how failure to look at continuous data can lead to self-imposed confusion and obfuscation, also cost real money, and delay important projects.

As the resident expert in clinical diagnostic assays in a large pharmaceutical company (that’s not saying a great deal when the company didn’t nurture such expertise), I was pulled into an apparent assay issue impinging on an oncology clinical trial. The assay measured gene copy number (GCN) for a specific gene; a GCN value above a specified cutoff was a study enrollment criterion. That is, it was a companion diagnostic assay (CDx) for the therapy under study. Two labs carried out testing for the study, each serving a different geographic region.

Recently there had been some operational issues with the assay which had required troubleshooting; the assay vendor had confirmed the issue, put a fix in place, and, for good measure, both labs repeated operator proficiency validations. Then patient screening for the study resumed. After some time, however, the trial team noticed that the “prevalence” (incidence of GCN over cutoff) was higher at one lab than the other. It was decided to pause study enrollment once again. zNote that for a pharmaceutical company, completing trials quickly is the coin of the realm; pausing a trial was a Very Serious Matter. Meetings were held, numbers were compiled, and still bigger meetings were held. Finally this expanding process grew to include a bystander previously unaware of the entire study, i.e., me.

When I joined my first meeting, it was chaired by the head of Oncology, which, for the company organization at the time, reported to the CEO. Lots of highly-paid senior people were there; this was one of those meetings where the cash clock ticked quickly.

I was given the information that had been compiled up to then. This included assay positivity counts at each site. I asked, “Where are the GCN numbers for each of the sites? What does the GCN distribution look like at each site?”

Such information had not been compiled!

Consider for a moment what this indicates about priorities and corporate culture. The company was expending significant resources, pausing a trial and using a top executive’s time. It would have been the simplest thing to organize GCN values–they were available in the clinical database, waiting to be visited. There’s no question that the values actually measured would give a more complete representation than the processed values. Yet this value was not widely shared. If this describes your organization too, you have work to do!

As an aside, there’s another aspect to this: when you’re troubleshooting a data-related issue, investigate the data process from first information acquisition to final result before you invest time in a lot of other approaches. At what point does the data begin to look anomalous? Or, if you prefer, start at the end and work towards the beginning. The point is to be systematic and to “scan” the whole process to come to an understanding of the state of things. Looking at continuous data often means looking upstream; I also have an expensive war story about this. The error is committed again and again.

But back to our story: in not much time the team pointed me to GCN numbers and I was able to determine that the clinical cutoff for GCN was very near the median GCN value. This is not necessarily an error, but it is definitely problematic: a small shift in the distribution of measured values, such as can easily happen with many assays, will induce a substantial change in positivity rate. I fitted density estimates; they had similar shapes but one was shifted slightly higher than the other. If I shifted the higher density down by the difference in medians, the densities lined up quite well, and furthermore the apparent positivity rates closely agreed.

This difference in median was smaller than the noise in the assay, so an assay scientist wouldn’t worry too much about it, and would certainly recommend against placing the assay in a context where a trivial change (relative to assay variability) would be interpreted gravely. Red flags and warnings should have gone up when the clinical cutoff was suggested.

The team decided to carry out a paired sample study. Regulatory requirements prevented either lab from sending clinical samples to the other lab, but the assay vendor could split samples, test them, and send them to each lab. Then we could compare each lab to the vendor. Long story short, it turned out there was a small difference between the vendor and both labs, and in the same direction, but this difference was not meaningful. The trial resumed. Frankly, while there was an observed difference, when interpreted with quantitative data and an understanding of variability, there was no real issue. By looking at processed data and not turning quickly to the underlying quantities when questions arose, the team had gotten themselves in very costly tizzy.

Here’s what I’ve taken away from these experiences:

For more on this topic, check out Frank Harrell Jr.’s essays How to Do Bad Biomarker Research and a chapter on Information Loss.

#continuousdata #dataanalysis #statistics