Causal bias, statistical bias, and estimands confusion

The term ‘statistical estimand’ just needs to go

What do ‘Causal bias’ and ‘statistical bias’ mean?

A few years ago I had an ohhh! moment when I heard someone explain what someone else meant by the term causal bias. Causal bias was being used to denote the discrepancy between the value of an (incorrect) identifying expression and the desired function of a counterfactual distribution. Let’s firm things up and say we want the expectation, then in not-words, causal bias might be written:

\(\text{Causal bias}= \sum_x E(Y|Z=z,X)P(X)-E(Y^z) \qquad (1)\)

So far so good. The helpful explaining person then described statistical bias as the discrepancy between the expected value of the estimator and the true value of the identifying expression, e.g.:

\(\text{Statistical bias (purportedly)}= E(\widehat{\theta})-\sum_x E(Y|Z=z,X)P(X) \qquad (2)\)

The bias of an estimator is usually defined as:

\(\text{Bias}=E(\widehat{\theta})-\theta \qquad (3)\)

where θ denotes the true value of the estimand (can’t believe I’ve managed not to say the word estimand until now).

For me and the people I interact with day-to-day, the term estimand refers to a contrast of counterfactual distributions under alternative actions (or potential outcomes; I’m using them interchangeably here… will probably get yelled at for my ignorance). Taking this, the usual definition of bias in (3) is different to that in (2). In (3), θ relates to a function of a counterfactual distribution but in (2) it relates to the identifying expression.

Confusion and miscommunication between camps

For the purpose of estimating causal effects, a standard flow you might see written out goes:

Estimand → Identifying expression → Estimator → Estimate (including uncertainty).

A key idea is to separate causal from statistical inference (which seems a useful separation). Putting words over the arrows:

\(E(Y^z) \xrightarrow{\text{Causal}} \sum_x E(Y|Z=z,X)P(X) \xrightarrow{\text{Statistical}} \widehat{\theta} \xrightarrow{\text{Computer goes } brrrrm} 1 [0,2]\)

Some people use slightly different terminology: they call the estimand the causal estimand and the identifying expression the statistical estimand – who the hell thought this is helpful terminology! (A reliable source tells me the folks who do the ‘Causal roadmap’ are to blame.) Needless to say, I think using estimand twice in this way is extremely unhelpful and has led to all confusion. In particular, some causal inference folks have been critical of the ICH E9(R1) Addendum. Some fun criticisms I’ve heard

• (Reasonable) ‘It doesn’t use the word causal despite using causal concepts throughout.’

• (Ignorant) ‘It doesn’t cite a paper I like’ – indeed, these documents to not cite references; maybe find out what you’re commenting on?

• (Baffling) ‘This document on estimands doesn’t mention g-formula!’ (for example). People who like the addendum are baffled by this and think, ‘Well of course it doesn’t, it’s about estimands and g-formula is about estimation.’ I think the above helps me see the source of the confusion. The addendum is about describing estimands using words.

Does it matter in methods work?

Suppose we are running a simulation study and are interested in bias of an estimator. Typically we use expression (3) in calculating bias. In any case, (2) and (3) will usually amount to the same thing.

What about when they are different, i.e. there is causal bias? Why should we be interested in (2) rather than (3)? The bias defined in (2) seems useful as a way of decomposing (3) so we can quantify the relative contribution of causal and statistical bias, but I think I’d rather know the overall / accumulated bias.

Non-causal estimands

In my circles, when people say estimand, they usually mean causal estimand. Of course I’m aware that there are non-causal estimands, but I don’t usually think about them.

In surveys, the estimand might be some proportion in a well-defined population, and that doesn’t rely on counterfactuals. In that case it seems uncontroversial to use the (analogue-to) expression (2) to define bias.

Predictimands are close to causal, but the vanilla version is a counterfactual distribution, but without a contrast given alternative actions.