Explorations of probabilistic thinking

This blog goes out to people who are not professional statisticians or data analysts, but who need to work with one. Or hire one. If this describes you, here’s something very important: you need to know about archetypes of statistical practice.

In particular, if you need support for a problem that does not directly involve clinical trials, you should be discerning of what skills you need, and you can’t assume that just any statistician will do.

So, what in the world is an archetype of statistical practice?

In my nearly-30-year career as a statistician working in industry, it’s struck me that there are three main patterns of statistical work: – The clinical statistician – The industrial statistician – The machine-learning or algorithmic statistician

These are work areas, which call for certain work practices or mental orientations. Any specific statistician naturally adopts the practice pattern for their area of work, or they naturally migrate to the work that fits their orientation pattern. I call these “archetypes” of statistical orientation.

A person can have more than one archetype, and some may be able to adapt their work pattern to the different work at hand. But some strongly live in one area predominately, and would perform poorly in a different context, if they don’t receive specific training.

You may be asking, “Where do data scientists fit into this system?” I first came up with this system long ago, before “data scientist” was a thing, so humor me and let me parse the statistical world first, and then I’ll make an attempt at placing data scientists into it.

A caveat: my career experience has been in the health field (diagnostic devices and, a bit, pharmaceuticals). If you’re coming from a different background, perhaps things appear a bit different to you? I’d be interested to hear your thoughts.

With that, the three archetypes follow.

The clinical statistician

The clinical statistician supports clinical studies interpreted in an inherently adversarial context, whether they’re supporting a research manuscript for publication or a drug application to health authorities. They make arguments concerning the appropriate degree of evidence contained in the data for or against a hypothesis, sufficient to convince a skeptical audience. Therefore they must make sure everything is in order and all critical assumptions are met.

“Biostatistics” is the subfield of statistics that focuses on clinical trials. There are challenges that arise uniquely in clinical trials, and general statisticians may not be that familiar with some of them.

To illustrate a situation in which argument is key, and it’s critical to not make mistakes: in graduate school I took a math course called “Real Analysis”, which is the theoretical development of calculus. For one exam we were given one hour to prove 10 statements. In that hour I finished proving three of them; for the others I wrote down some thoughts but did not fully prove the statements. I was careful to say what I was confident in and why.

For confidently proving only 30% of the statements, I earned an “A” on the exam. I demonstrated that I was not extremely creative in math, but I knew what followed from what, and the latter is a valuable skill. In fact, mathematics is an edifice in which one true statement is based on another, so it strongly depends on there being no wrong things in the framework. This carries over to arguing for a skeptical audience: say what you can, don’t say what you can’t, and above all, don’t say a wrong thing. A wrong thing could cause your whole case to crumble. And if they find you saying one wrong thing, and you’re unaware that it’s wrong, can they believe other things you’re saying?

I’ve seen clinical statisticians look at a careful post-hoc exploratory analysis (which can be critical to the business) and make a blanket statement that statistical inference in exploratory analyses can’t be relied upon. This is true, but quite unhelpful–more helpful would be: how compromised is the particular exploratory analysis? This is one example where an orientation is not wrong but inappropriate to the context.

The industrial statistician

The industrial statistician focuses on experiments with many controllable factors, such as in a lab experiment. Their goal is to optimize a product or process, or to develop a predictive model of a system. Here it is critical to determine the most important contributors to the process and to understand different sources of variability. Once the most important factors are identified and characterized, less-important factors are of little interest. Assessing evidence for relationships sufficient to convince a skeptical audience is not important; what is important instead is reaching findings that will move the project forward usefully, even if they’re imperfect approximations of reality.

There is a technology for handling a large number of predictor variables under experimental control, especially where every single observation is expensive (e.g., it requires an entire run of a pilot manufacturing line).

For instance, the field of “fractional factorial designs” can develop a useful local model for 5 controlled factors by collecting only 9 runs (8 runs that perturb all 5 factors and one run in the nominal “center” in order to detect whether a linear model is adequate); an additional 10th run at the center point again would be valuable in order to estimate pure error. This assumes that interactions involving 3 factors or more are trivial, which is realistic in most cases. 9 or 10 runs stands in contrast to the 2⁵ = 32 runs one might naïvely expect as a minimal perturbation of 5 factors (not counting center points or replication). 32 runs will support estimation of all possible interactions up to 5-way, but it is highly, highly unlikely that all such interactions will be active. Finding a balanced subset of runs in 5-dimensional space is a nontrivial exercise, and a geometrically appealing one. This is a useful skill, and one that a clinical statistician could very well have no awareness of.

There is an analogy with numerical optimization routines. Note that such algorithms work iteratively, and at each iteration they make a local linear or quadratic approximation to the function of interest. It isn’t critical that the function be truly linear or quadratic; the approximation only needs to be good enough to move the search process forward. The same is true for modeling of experiment data: the model needs to be a good enough local approximation that it moves the project forward; it need not be correct.

In fact, industrial work is usually iterative, in its best form. In fact, a rule of thumb is to spend no more than 20% of your budget on your first experiment, because you fully expect subsequent experiments. Subsequent experiments can incorporate previous findings, and it is efficient not to waste resources on factors or estimates that, given prior data, can be neglected. In a sequential context, correctness is less important than in the clinical context, provided results are correct enough; after all, if the conclusions of one experiment are a little off, the next experiment will refine them.

There is an art to working with a team to elicit a list of all potential factors, develop a strategy to handle them, and to bring the team on board with the experimental design. Thus there is a bit that is ineffable and social in the practice of industrial statistics.

The machine-learning or algorithmic statistician

Statistics is about learning from data, and standard statistical methods develop this learning by making assumptions that might be more useful than true. Do errors follow a Gaussian (normal) distribution? Are relationships linear? And we begin to realize that these assumptions, while convenient, are not actually required or even the main idea. We can do about as well if we assume that a relationship is smooth, rather than linear, for instance. This direction of interest leads to semiparametric modeling, multivariate clustering, semiparametric density estimation, and predictive models that allow for arbitrary complexity of interaction (neural nets, random forest, support vector machines). This flavor of statistician also takes responsibility for assessing a model’s generalizability to future data.

However, the big dichotomy in statistics is between clinical and industrial; the machine-learning orientation is usually added to one of the others, when it is expressed at all.

Data scientists and statisticians

Where is the boundary between data science and statistics? Is there one? Much ink has been spilled on this so it is probably foolish to pursue it here…but what the heck, completeness demands it. My own experience (with clinical biomarker exploration) suggests the following: – Data scientists are intentional about the craft of programming and managing data. While a few statisticians have intentionally nurtured their craft, the community doesn’t see it as a universal need. A randomly-selected data scientist is likely to be a better programmer than a randomly-selected statistician. – Data scientists take ownership of data pipelines and data handling, more so than statisticians. – Statisticians own the question of inference. If you’re making statistical inference, you’re doing statistics. Whether or not you consider yourself a statistician. – Data scientists tend to take more responsibility than statisticians for understanding the scientific background. – I see a bifurcation in the data science community: there are those whose analysis process is to use loops and a small set of hypothesis tests and plots, then interpret the pile of results that result. Others adopt machine-learning and clustering methods eagerly.

What does this all mean?

In the health-related industry, most statisticians work in the clinical area, and support the clinical archetype. This is very important work with its own idiosyncracies, and it’s good that these practitioners generally adopt the archetype appropriate to the work. Some of these practitioners may also be able to adopt one or more of the other archetypes when placed into a different context, while others may not.

The proportion of clinical statisticians among statisticians in health-related areas is so high that many in the industry don’t realize that there is anything else. They refer to all statisticians as “biostatisticians” and expect that clinical statisticians can address any statistical need. This is a big error and can lead to business and project issues.

It would be just as big of an error to place one of the other archetypes into a clinical role if they cannot adopt the clinical orientation. However, given that the clinical role is so prevalent and highly developed, this direction of error rarely happens, or at least it is caught right away. It is the clinical-to-other direction that is more likely to be undetected, and to lead to problems. All it takes is a blasé pointy-haired boss to put the wrong person into place and the stage for mayhem is set.

In essence, managers in health-related organizations need to know that not all statisticians are alike.