Tim this is super helpful. II was not aware of Wood’s paper or yours. I’ll read both. I hope that Chan & Meng performs better but time will tell. I wish they had answered my email. When authors write a paper and then disappear it’s hard to make their research pay off.
Nice post. I’m interested in reading more about the Bayesian perspective. I think we may need to retire Rubin’s rule and go Bayesian because of inaccuracies in confidence coverage that result in assuming either that the overall effect estimate has a normal distribution or has a t distribution with degrees of freedom approximated by another formula of Rubin. Multiple imputation results in heavier-than-normal tails of the sampling distribution of an estimator, even when the model is purely Gaussian. By conducting separate Bayesian analysis for each completed dataset and using posterior stacking to get an overall posterior distribution, this problem largely takes care of itself. But concerning the original goal of your analysis I wonder if formal Bayesian modeling can deal with the particular type of MNAR you’re addressing.
Thanks Frank! I confess that I’ve only recently stumbled across the idea of Bayesian analysis after multiple imputation (with thanks to Harlan Campbell and Jeremy Saxe), and wish I’d known about it earlier. It initially seemed mad (why not just go fully Bayesian). But I think it’s practicable and elegant to do this MI-then-Bayes. Drawing the imputations using MI is a bit ragged but straightforward. The Bayesian inference at the end is so simple and elegant, the way it sidesteps the need to estimate within- and between-imputation variance, and approximate degrees-of-freedom for the t-distribution. Do you know of any references on this besides Zhou and Reiter, 2010 (https://www.tandfonline.com/doi/abs/10.1198/tast.2010.09109) and Gelman’s Bayesian Data Analysis book? Thanks.
Regarding doing reference-based analysis in a fully-Bayesian way, I think this would suffer from self-inefficiency as well, and by doing the imputation and analysis jointly, the posterior would be too narrow. I’d be delighted to discover I’m wrong about this – please say! So the way MI separates imputation from analysis here is, I think, how we get information-anchoring.
Thank you so much for that reference. I've been looking for the prime reference for posterior stacking. I don't know of another reference except for the one that uses posterior stacking for getting likelihood ratio tests - see https://hbiostat.org/rmsc/missing#sec-missing-lrt . I think you're right about Bayes not solving the other problem. I'm hoping there is an explicit Bayesian modeling for handling that. Sometimes such things are handled by fitting joint models as I did at https://fharrell.com/post/hxcontrol . I think that full Bayesian modeling of missing values is preferable but I don't know of guidance for actually making that practical often enough. Because of that multiple imputation fed into Bayesian posterior stacking is expected to have a long shelf life.
Sorry I failed to reply to this Frank. Agree with your comments on the principle vs. practicalities of Bayesian approaches to handling missing data. I had not come across Chan and Meng (2022) before and must read it. They seem to have missed Angela Wood's paper (https://doi.org/10.1002/sim.3177). There, and in a later paper I wrote (https://doi.org/full/10.1002/sim.6553), likelihood ratio tests based on stacking showed disappointing performance compared with Wald tests based on Rubin's rules. A quick look suggests Chan & Meng is a bit different, so hopefully it performs better, though I'd like to understand how the two relate. Thanks again!
Tim this is super helpful. II was not aware of Wood’s paper or yours. I’ll read both. I hope that Chan & Meng performs better but time will tell. I wish they had answered my email. When authors write a paper and then disappear it’s hard to make their research pay off.
Nice post. I’m interested in reading more about the Bayesian perspective. I think we may need to retire Rubin’s rule and go Bayesian because of inaccuracies in confidence coverage that result in assuming either that the overall effect estimate has a normal distribution or has a t distribution with degrees of freedom approximated by another formula of Rubin. Multiple imputation results in heavier-than-normal tails of the sampling distribution of an estimator, even when the model is purely Gaussian. By conducting separate Bayesian analysis for each completed dataset and using posterior stacking to get an overall posterior distribution, this problem largely takes care of itself. But concerning the original goal of your analysis I wonder if formal Bayesian modeling can deal with the particular type of MNAR you’re addressing.
Thanks Frank! I confess that I’ve only recently stumbled across the idea of Bayesian analysis after multiple imputation (with thanks to Harlan Campbell and Jeremy Saxe), and wish I’d known about it earlier. It initially seemed mad (why not just go fully Bayesian). But I think it’s practicable and elegant to do this MI-then-Bayes. Drawing the imputations using MI is a bit ragged but straightforward. The Bayesian inference at the end is so simple and elegant, the way it sidesteps the need to estimate within- and between-imputation variance, and approximate degrees-of-freedom for the t-distribution. Do you know of any references on this besides Zhou and Reiter, 2010 (https://www.tandfonline.com/doi/abs/10.1198/tast.2010.09109) and Gelman’s Bayesian Data Analysis book? Thanks.
Regarding doing reference-based analysis in a fully-Bayesian way, I think this would suffer from self-inefficiency as well, and by doing the imputation and analysis jointly, the posterior would be too narrow. I’d be delighted to discover I’m wrong about this – please say! So the way MI separates imputation from analysis here is, I think, how we get information-anchoring.
Thank you so much for that reference. I've been looking for the prime reference for posterior stacking. I don't know of another reference except for the one that uses posterior stacking for getting likelihood ratio tests - see https://hbiostat.org/rmsc/missing#sec-missing-lrt . I think you're right about Bayes not solving the other problem. I'm hoping there is an explicit Bayesian modeling for handling that. Sometimes such things are handled by fitting joint models as I did at https://fharrell.com/post/hxcontrol . I think that full Bayesian modeling of missing values is preferable but I don't know of guidance for actually making that practical often enough. Because of that multiple imputation fed into Bayesian posterior stacking is expected to have a long shelf life.
Sorry I failed to reply to this Frank. Agree with your comments on the principle vs. practicalities of Bayesian approaches to handling missing data. I had not come across Chan and Meng (2022) before and must read it. They seem to have missed Angela Wood's paper (https://doi.org/10.1002/sim.3177). There, and in a later paper I wrote (https://doi.org/full/10.1002/sim.6553), likelihood ratio tests based on stacking showed disappointing performance compared with Wald tests based on Rubin's rules. A quick look suggests Chan & Meng is a bit different, so hopefully it performs better, though I'd like to understand how the two relate. Thanks again!