Fixed vs. repeatedly simulated complete data in missing data simulation studies

I can almost guarantee the correct answer for you is ‘repeatedly-simulated’

I’ve just had a letter-to-the-editor published in Biometrical Journal. We consider whether we researchers should fix or simulate the complete data in simulation studies that evaluate missing methods for handling missing data. Our view is that it’s very rarely appropriate to fix the complete data.

So this is a letter-to-the-editor… and, unusually for a letter, I like loads about the paper it’s commenting on: ‘Towards a standardized evaluation of imputation methodology’ by Hanne Oberman and Gerko Vink. I was a reviewer of their paper, and we just could not see eye-to-eye on this particular point. They find it convenient to ‘fix’ a complete dataset as the truth and repeatedly simulate missingness. Essentially they regard this as a convenient way to remove a nuisance source of variation.

My view is very different: a central aim of analysis with missing data is to estimate all source of variation. When we remove a source of variation from our simulation study, we can’t properly assess whether a given method does this! After a few back-and-forths, I said to the editor that we were not going to agree. They helpfully suggested that the authors publish their paper with their view and I write a letter detailing mine.

Now, I’ve come across this idea several times in the past and always ended up arguing against it. Having discussed the issue with various collaborators in the past, some pretty extensively, I invited them join me as authors (nice to have this publication with several of my favourite missing data collaborators all at once).

I’m not going to list all our reasons here – you can read the letter.

I did design a cheeky little simulation study to show how the ‘fixed complete data’ approach falls apart when you use a superefficient imputation procedure, reported in the letter. Hey, I’ve never reported a simulation study before where the comparison of central interest was between alternative data-generating mechanisms!

So myself and Tra My Pham are Biom J’s Stata code reviewers. But we were both coauthors. Our predecessor was Ian White… also a coauthor. In the end, Julian Lange reviewed my code. He discovered something very strange, that when he changed the seed under the ‘fixed’ approach, he got very different performance. I hadn’t detected this, and it took a 2h conversation with Ian White to understand what was going on, but was a really important point! This resulted in me extending the simulation study (several seeds) and being able to explain this odd result. Big thumbs up for code review.

I’m now looking forward to reading Oberman & Vink’s response to our letter.