Mosaic Monte Carlo: A New Method of Simulation Design to Improve the Generalizability of Findings
DOI:
https://doi.org/10.35566/jbds/gomer2026Keywords:
Simulation design, Monte Carlo, GeneralizabilityAbstract
Monte Carlo simulation studies are an essential tool to test the performance of statistical methods. They are often implemented by generating data from a small number of data-generating models for a large number of replications. However, it is not guaranteed that a statistical method tested on a handful of data-generating models will perform well in other scenarios. Simulations are necessarily limited in scope, and so it is all too possible for results to unknowingly fail to generalize to real applications. This issue of generalizability is particularly relevant for methods that are more sensitive to parameter values such as those used in missing data analysis and Bayesian statistics. In this paper, we propose a new type of simulation design called Mosaic Monte Carlo that can help improve the generalizability of Monte Carlo simulation studies to real world applications. This method implements simulations by breaking up replications into smaller subsets, each using a different data-generating model. This approach to simulation design improves the generalizability of results beyond traditional designs.
References
Boulesteix, A.-L., Groenwold, R. H., Abrahamowicz, M., Binder, H., Briel, M., Hornung, R., … Panel, S. (2020). Introduction to statistical simulations in health research. BMJ Open, 10(12), e039921. doi: https://doi.org/10.1136/bmjopen-2020-039921
Boulesteix, A.-L., Lauer, S., & Eugster, M. J. A. (2013). A plea for neutral comparison studies in computational sciences. PLOS ONE, 8(4), e61562. doi: https://doi.org/10.1371/journal.pone.0061562
Boulesteix, A.-L., Stierle, V., & Hapfelmeier, A. (2015). Publication bias in methodological computational research. Cancer Informatics, 14(S5), 11–19. doi: https://doi.org/10.4137/CIN.S30747
Brooks, C. (2002). Introductory economics for finance. Cambridge University Press.
Burton, A., Altman, D. G., Royston, P., & Holder, R. L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25(24), 4279–4292. doi: https://doi.org/10.1002/sim.2673
Casella, G., & Berger, R. L. (2001). Statistical inference (2nd ed.). Duxbury.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351. doi: https://doi.org/10.1037/1082-989X.6.4.330
Curran, P. J., Bollen, K. A., Paxton, P., Kirby, J., & Chen, F. (2002). The noncentral chi-square distribution in misspecified structural equation models: Finite sample results from a monte carlo simulation. Multivariate Behavioral Research, 37(1), 1–36. doi: https://doi.org/10.1207/s15327906mbr3701_01
Franklin, J. M., Schneeweiss, S., Polinski, J. M., & Rassen, J. A. (2014). Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases. Computational Statistics & Data Analysis, 72, 219–226. doi: https://doi.org/10.1016/j.csda.2013.10.018
Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F., & Hothorn, T. (2020). mvtnorm: Multivariate normal and t distributions [Computer software manual]. Retrieved from https://CRAN.R-project.org/package=mvtnorm (R package version 1.1-1)
Gomer, B., Jiang, G., & Yuan, K.-H. (2019). New effect size measures for structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 26(3), 371–389. doi: https://doi.org/10.1080/10705511.2018.1545231
Gomer, B., & Yuan, K.-H. (2021). Subtypes of the missing not at random missing data mechanism. Psychological Methods, 26(5), 559–598. doi: https://doi.org/10.1037/met0000377
Goutelle, S., Bourguignon, L., Maire, P. H., Van Guilder, M., Conte, J. E., & Jelliffe, R. W. (2009). Population modeling and monte carlo simulation study of the pharmacokinetics and antituberculosis pharmacodynamics of rifampin in lungs. Antimicrobial Agents and Chemotherapy, 53(7), 2974–2981. doi: https://doi.org/10.1128/AAC.01520-08
Hu, L.-T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. doi: https://doi.org/10.1080/10705519909540118
Iranmanesh, H., Parchami, A., & Sadeghpour Gildeh, B. (2022). Statistical testing quality and its monte carlo simulation based on fuzzy specification limits. Iranian Journal of Fuzzy Systems, 19(3). doi: https://doi.org/10.22111/ijfs.2022.6940
Ke, Z., & Wang, L. (2015). Detecting individual differences in change: Methods and comparisons. Structural Equation Modeling: A Multidisciplinary Journal, 22(3), 382–400. doi: https://doi.org/10.1080/10705511.2014.936096
Koehler, E., Brown, E., & Haneuse, S. J.-P. A. (2009). On the assessment of monte carlo error in simulation-based statistical analyses. The American Statistician, 63(2), 155–162. doi: https://doi.org/10.1198/tast.2009.0030
Kulinskaya, E., Hoaglin, D. C., & Bakbergenuly, I. (2021). Exploring consequences of simulation design for apparent performance of methods of meta-analysis. Statistical Methods in Medical Research, 30(7), 1667–1690. doi: https://doi.org/10.1177/09622802211013065
Leigh, J. W., & Bryant, D. (2015). Monte carlo strategies for selecting parameter values in simulation experiments. Systematic Biology, 64(5), 741–751. doi: https://doi.org/10.1093/sysbio/syv030
McKay, M. D., Beckman, R. J., & Conover, W. J. (2000). A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 42(1), 55–61. doi: https://doi.org/10.1080/00401706.2000.10485979
Morris, T. P., White, I. R., & Crowther, M. J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11), 2074–2102. doi: https://doi.org/10.1002/sim.8086
Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte carlo experiments: Design and implementation. Structural Equation Modeling: A Multidisciplinary Journal, 8(2), 287–312. doi: https://doi.org/10.1207/s15328007sem0802_7
Preecha, C. (2004). Numbers of replications required in anova simulation studies. University of Northern Colorado.
R Core Team. (2020). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from https://www.R-project.org/
Schaffer, J. R., & Kim, M.-J. (2007). Number of replications required in control chart monte carlo simulation studies. Communications in Statistics—Simulation and Computation, 36(5), 1075–1087. doi: https://doi.org/10.1080/03610910701539963
Schreck, N., Slynko, A., & Saadati, M. (2024). Statistical plasmode simulations—potentials, challenges and recommendations. Statistics in Medicine, 43(9), 1804–1825. doi: https://doi.org/10.1002/sim.10012
Siepe, B. S., Bartoš, F., Morris, T. P., Boulesteix, A.-L., Heck, D. W., & Pawel, S. (2024). Simulation studies for methodological research in psychology: A standardized template for planning, preregistration, and reporting. Psychological Methods. doi: https://doi.org/10.1037/met0000695
Skrondal, A. (2000). Design and analysis of monte carlo experiments: Attacking the conventional wisdom. Multivariate Behavioral Research, 35(2), 137–167. doi: https://doi.org/10.1207/s15327906mbr3502_1
Spence, I. (1983). Monte carlo simulation studies. Applied Psychological Measurement, 7(4), 405–425. doi: https://doi.org/10.1177/014662168300700403
Stevens, J. P. (1992). Applied multivariate statistics for the social sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
Supawan, P. (2004). An examination of the number of replications required in regression simulation studies. University of Northern Colorado.
Tuffin, B. (1996). On the use of low discrepancy sequences in monte carlo methods (No. 1060). Rennes, France: IRISA.
