Causal bias, statistical bias, and estimands confusion
The term ‘statistical estimand’ just needs to go
What do ‘Causal bias’ and ‘statistical bias’ mean?
A few years ago I had an ohhh! moment when I heard someone explain what someone else meant by the term causal bias. Causal bias was being used to denote the discrepancy between the value of an (incorrect) identifying expression and the desired function of a counterfactual distribution. Let’s firm things up and say we want the expectation, then in not-words, causal bias might be written:
So far so good. The helpful explaining person then described statistical bias as the discrepancy between the expected value of the estimator and the true value of the identifying expression, e.g.:
The bias of an estimator is usually defined as:
where θ denotes the true value of the estimand (can’t believe I’ve managed not to say the word estimand until now).
For me and the people I interact with day-to-day, the term estimand refers to a contrast of counterfactual distributions under alternative actions (or potential outcomes; I’m using them interchangeably here… will probably get yelled at for my ignorance). Taking this, the usual definition of bias in (3) is different to that in (2). In (3), θ relates to a function of a counterfactual distribution but in (2) it relates to the identifying expression.
Confusion and miscommunication between camps
For the purpose of estimating causal effects, a standard flow you might see written out goes:
Estimand → Identifying expression → Estimator → Estimate (including uncertainty).
A key idea is to separate causal from statistical inference (which seems a useful separation). Putting words over the arrows:
Some people use slightly different terminology: they call the estimand the causal estimand and the identifying expression the statistical estimand – who the hell thought this is helpful terminology! (A reliable source tells me the folks who do the ‘Causal roadmap’ are to blame.) Needless to say, I think using estimand twice in this way is extremely unhelpful and has led to all confusion. In particular, some causal inference folks have been critical of the ICH E9(R1) Addendum. Some fun criticisms I’ve heard
(Reasonable) ‘It doesn’t use the word causal despite using causal concepts throughout.’
(Ignorant) ‘It doesn’t cite a paper I like’ – indeed, these documents to not cite references; maybe find out what you’re commenting on?
(Baffling) ‘This document on estimands doesn’t mention g-formula!’ (for example). People who like the addendum are baffled by this and think, ‘Well of course it doesn’t, it’s about estimands and g-formula is about estimation.’ I think the above helps me see the source of the confusion. The addendum is about describing estimands using words.
Does it matter in methods work?
Suppose we are running a simulation study and are interested in bias of an estimator. Typically we use expression (3) in calculating bias. In any case, (2) and (3) will usually amount to the same thing.
What about when they are different, i.e. there is causal bias? Why should we be interested in (2) rather than (3)? The bias defined in (2) seems useful as a way of decomposing (3) so we can quantify the relative contribution of causal and statistical bias, but I think I’d rather know the overall / accumulated bias.
Non-causal estimands
In my circles, when people say estimand, they usually mean causal estimand. Of course I’m aware that there are non-causal estimands, but I don’t usually think about them.
In surveys, the estimand might be some proportion in a well-defined population, and that doesn’t rely on counterfactuals. In that case it seems uncontroversial to use the (analogue-to) expression (2) to define bias.
Predictimands are close to causal, but the vanilla version is a counterfactual distribution, but without a contrast given alternative actions.
"The bias defined in (2) seems useful as a way of decomposing (3) so we can quantify the relative contribution of causal and statistical bias, but I think I’d rather know the overall / accumulated bias."
Wouldn't the bias in (2) be a potentially useful indicator of how much model misspecification contributes to overall bias? Perhaps a signal of how useful non/semi-parametric estimators might be relative to improving design or data quality?
A couple of postscripts on this:
1. There’s a paper in American Sociological Review that uses the terms ‘theoretical estimand’ (for estimand) and ‘empirical estimand’(for identifying expression): https://journals.sagepub.com/doi/10.1177/00031224211004187
My gripes above about ‘statistical estimand’ directly translate to ‘empirical estimand’, but that paper gets an extra gripe point for adding the word ‘theoretical’ to ‘estimand’. This makes it sound somehow irrelevant, when it’s what I’d term the ‘actual estimand’.
2. So far three people have commented ‘You haven’t actually defined your estimand in this post’. No shit Sherlock! I think my point is for whatever the causal estimand is. I feel like there must be something I’m missing, because it’s been made by three people so far. OTOH perhaps they discussed it so the comments aren’t independent and it’s more like one.
3. I’ve been painfully aware recently that I sometimes use ’estimand‘ in the sense of ‘statistical estimand’ above, when it’s not for a causal estimand. E.g. in our letter-to-the-editor about Oberman & Vink’s paper (blog post at https://open.substack.com/pub/tpmorris/p/fixed-vs-repeatedly-simulated-complete). So I appreciate that, since I could argue against myself about this, others could. I frequently hear from applied researchers that they regard causal estimands as alchemy, enough so that we really need to minimise confusion, and that’s why I’ve come out swinging above.