I didn’t wake up this morning planning to use a Backstreet Boys lyric as a post title but here we are.
For the majority of research documents I comment on (and research publications I read), I make the same comment several times:
This says what you’ve done, but not why.
– Me (2007 onwards)
For any method that actually got used, there would have been several other methods (or variations on the one used) that might have been picked. I’m not impressed by the whole ‘let’s just do them all’ multiverse thing. Instead, I want to know what led to the particular choice you made.
By not saying why you picked a particular method, research appears flimsier than it otherwise might. It makes the reader assume your rationale was so weak you were embarrassed to give it; something like ‘it was just what I thought of’, ‘I have code to do this’ or ‘I have done it this way before (or someone else has)’. The thing is, when pushed, people often have a good rationale that buttresses their choice and so their conclusions. So researchers’ aversions to explaining why really baffles me.
One example is complete-case analysis in the presence of missing data. It’s often viewed as a cop-out (you just did it because it’s easy), but it’s valid and sometimes efficient under certain assumptions about missingness. When people say ‘we did complete-case analysis’, I will just assume the former. But when I say this, the response is sometimes really a convincing complete-case case.
For methodology work, I’ve often pointed people to our phases paper1. For example, some people labour under the misconception that every simulation study must have a ‘realistic’ data-generating mechanism. Sometimes the answer to this is that we’re not yet at late-phase, when we would want ‘realistic’; we’re actually at the earlier phase of knocking out candidate methods that don’t work even in simple settings, and suggesting that the remainder are taken forward for fuller evaluation.
Unfortunately, some people who actually do research have a naïve pop-sci-reader’s view of scientific research as a purely objective venture that reveals truths about the world. If you are this kind of person, saying why you used a certain method is a sign of weakness in admitting your own role (‘Oh no, the results depend on choices made by the researcher!’). Actually it looks less subjective if you can justify the choices. (More importantly, can we please get over this ridiculous, superficial view of scientific research as ‘objective’?)
This isn’t going to go away any time soon and clearly I’m going to have to keep saying tell me why over and over ’til I die. In writing this post I’m trying to be constructive… but perhaps I’m just demonstrating a commitment to wasting even more time than necessary on this gripe 😜
Next post will to be brought to you by ‘this described what the results were but the readers needs to know what they mean.’
G Heinze, A-L Boulesteix, M Kammer, TP Morris, IR White, for the Simulation Panel of the STRATOS initiative. Phases of methodological research in biostatistics—Building the evidence base for new methods. Biometrical Journal, 2024. 66, 2200222. https://doi.org/10.1002/bimj.202200222