Careless Responding in Daily Diary Research: Detection and Impact on Intensive Longitudinal Data Analyses
DOI:
https://doi.org/10.35566/jbds/wang2026Keywords:
Careless responding, Daily diary, Intensive longitudinal data, Adolescents, Attention checksAbstract
Careless responding is a concern in online surveys and poses a threat to data quality in intensive longitudinal research, which often relies on repeated online surveys. Despite its potential impact, little research has examined careless responding in adolescent daily diary research. This study investigated the prevalence, trends, and baseline correlates of careless responding using two daily attention check (ACQ) questions and response time data in a 21-day daily diary study of 311 youths aged 12 to 16. Average compliance (75.1%) and careless responding rates (3.4% to 18.1%) were comparable to those reported in recent meta-analyses with adult samples, supporting the feasibility of collecting 21-day daily diaries from adolescents. Response time screening using the common 2-second-per-item rule was less sensitive than ACQs in detecting careless responding in adolescent daily diaries. Participants failed the second ACQ more frequently than the first, highlighting both the value of multiple ACQs per assessment and the potential benefit of shorter surveys. Younger adolescents and those with higher state anger, more neighborhood problems, lower anger control, and poorer emotion regulation were more likely to fail ACQs. Sensitivity analyses using various subsamples after removing careless responding cases suggested that longitudinal analysis results may be inflated when careless responding is not addressed. Our findings underscore the importance of incorporating attention check and response time measures and conducting sensitivity analyses to examine the robustness of statistical conclusions in intensive longitudinal research with youth.
References
Alegría, M., Cruz-Gonzalez, M., Yip, T., Wang, L., Fang, Y., Park, I. J. K., €¦ Shrout, P. E. (2026). Protective factors of the longitudinal association between discrimination and mental health among mexican-origin youth. Journal of the American Academy of Child and Adolescent Psychiatry. doi: https://doi.org/10.1016/j.jaac.2026.01.012
Alegría, M., Cruz-Gonzalez, M., Yip, T., Wang, L., Park, I. J. K., Fukuda, M., €¦ Shrout, P. E. (2024). Yearly and daily discrimination-related stressors and mexican youth's mental health and sleep: Insights from the first wave of a three-wave family study. Journal of the American Academy of Child and Adolescent Psychiatry, 63(11), 1134-1148. doi: https://doi.org/10.1016/j.jaac.2023.12.010
Asparouhov, T., & Muthén, B. (2019). Latent variable centering of predictors and mediators in multilevel and time-series models. Structural Equation Modeling: A Multidisciplinary Journal, 26(1), 119-142. doi: https://doi.org/10.1080/10705511.2018.1511375
Asparouhov, T., & Muthén, B. (2021). Expanding the bayesian structural equation, multilevel and mixture models to logit, negative-binomial, and nominal variables. Structural Equation Modeling: A Multidisciplinary Journal, 28(4), 622-637. doi: https://doi.org/10.1080/10705511.2021.1878896
Bolger, N., & Laurenceau, J. P. (2013). Intensive longitudinal methods: An introduction to diary and experience sampling research. Guilford press.
Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218-229. doi: https://doi.org/10.1037/pspp0000085
Brunner, T. M., & Spielberger, C. D. (2009). The State Trait Anger Expression Inventory, Second Edition, Child Adolescent (STAXI-C/A). Psychological Assessment Resources.
Cinelli, C., & Hazlett, C. (2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1), 39-67. doi: https://doi.org/10.1111/rssb.12348
Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24(4), 385-396. doi: https://doi.org/10.2307/2136404
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19. doi: https://doi.org/10.1016/j.jesp.2015.07.006
Daniels, M. J., & Hogan, J. W. (2008). Missing data in longitudinal studies: Strategies for Bayesian modeling and sensitivity analysis. Chapman and Hall/CRC. doi: https://doi.org/10.1201/9781420011180
Eisele, G., Vachon, H., Lafit, G., Kuppens, P., Houben, M., Myin-Germeys, I., & Viechtbauer, W. (2022). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136-151. doi: https://doi.org/10.1177/1073191120957102
Feng, Y., Cheon, Y. M., Yip, T., & Cham, H. (2021). Multilevel IRT analysis of the Everyday Discrimination Scale and the Racial/Ethnic Discrimination Index. Psychological Assessment, 33(7), 637-651. doi: https://doi.org/10.1037/pas0000906
Gottfried, J. (2024). Practices in data-quality evaluation: A large-scale review of online survey studies published in 2022. Advances in Methods and Practices in Psychological Science, 7(2), 1-7. doi: https://doi.org/10.1177/25152459241236414
Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348-362. doi: https://doi.org/10.1037/0022-3514.85.2.348
Hamaker, E. L., & Wichers, M. (2017). No time like the present: Discovering the hidden dynamics in intensive longitudinal data. Current Directions in Psychological Science, 26(1), 10-15. doi: https://doi.org/10.1177/0963721416666518
Heron, K. E., Everhardt, R. S., McHale, S. M., & Smyth, J. M. (2017). Using mobile-technology-based ecological momentary assessment (EMA) methods with youth: A systematic review and recommendations. Journal of Pediatric Psychology, 42(10), 1087- 1107. doi: https://doi.org/10.1093/jpepsy/jsx078
Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299-311. doi: https://doi.org/10.1007/s10869-014-9357-6
Jones, A., Earnest, J., Adam, M., Clarke, R., Yates, J., & Pennington, C. R. (2022). Careless responding in crowdsourced alcohol research: A systematic review and meta-analysis of practices and prevalence. Experimental and Clinical Psychopharmacology, 30(4), 381- 399. doi: https://doi.org/10.1037/pha0000546
Jones, A., Remmerswaal, D., Verveer, I., Robinson, E., Franken, I. H., Wen, C. K. F., & Field, M. (2019). Compliance with ecological momentary assessment protocols in substance users: A meta-analysis. Addiction, 114(4), 609-619. doi: https://doi.org/10.1111/add.14503
Kam, C. C. S., & Chan, G. H.-h. (2018). Examination of the validity of instructed response items in identifying careless respondents. Personality and Individual Differences, 129, 83-87. doi: https://doi.org/10.1016/j.paid.2018.03.022
Kam, C. C. S., & Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512-541. doi: https://doi.org/10.1177/1094428115571894
Kung, F. Y. H., Kwok, N., & Brown, D. J. (2018). Are attention check questions a threat to scale validity? Applied Psychology, 67(2), 264-283. doi: https://doi.org/10.1111/apps.12108
Laurenceau, J. P., DiGiovanni, A. M., & Bolger, N. (2026). Intensive longitudinal methods: Toward a psychological science of daily life. Annual Review of Psychology, 77(5), 513- 541. doi: https://doi.org/10.1146/annurev-psych-040325-025418
Liu, X., & Wang, L. (2021). The impact of measurement error and omitting confounders on statistical inference of mediation effects and tools for sensitivity analysis. Psychological Methods, 26(3), 327-342. doi: https://doi.org/10.1037/met0000345
Loeber, R., Farrington, D. P., Stouthamer-Loeber, M., & Van Kammen, W. B. (1998). Antisocial behavior and mental health problems: Explanatory factors in childhood and adolescence. Psychology Press. doi: https://doi.org/10.4324/9781410602930
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455. doi: https://doi.org/10.1037/a0028085
Mehl, M. R., & Conner, T. S. (2012). Handbook of research methods for studying daily life. The Guilford Press.
Park, I. J. K., Wang, L., Li, R., Yip, T., Valentino, K., Cruz-Gonzalez, M., €¦ Alegría, M. (2024). A daily diary study of discrimination and distress in mexican-origin adolescents: Testing mediating mechanisms. Child Development, 95(5), 1754-1769. doi: https://doi.org/10.1111/cdev.14108
Permutt, T. (2016). Sensitivity analysis for missing data in regulatory submissions. Statistics in Medicine, 35(17), 2876-2879. doi: https://doi.org/10.1002/sim.6753
Robins, J. M., Rotnitzky, A., & Scharfstein, D. O. (2000). Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In Statistical models in epidemiology, the environment, and clinical trials (p. 1-94). New York, NY: Springer. doi: https://doi.org/10.1007/978-1-4612-1284-3_1
Stosic, M. D., Murphy, B. A., Duong, F., Fultz, A. A., Harvey, S. E., & Bernieri, F. (2024). Careless responding: Why many findings are spurious or spuriously inflated. Advances in Methods and Practices in Psychological Science, 7(1), 1-19. doi: https://doi.org/10.1177/25152459241231581
Ulitzsch, E., Nestler, S., Lüdtke, O., & Nagy, G. (2024). A screen-time-based mixture model for identifying and monitoring careless and insufficient effort responding in ecological momentary assessment data. Psychological Methods. Advance online publication. doi: https://doi.org/10.1037/met0000636
Ulitzsch, E., Viechtbauer, W., Lüdtke, O., Myin-Germeys, I., Nagy, G., Nestler, S., & Eisele, G. V. (2025). Investigating the effect of experience sampling study design on careless and insufficient effort responding identified with a screen-time-based mixture model. Psychological Assessment, 37(8), 347-359. doi: https://doi.org/10.1037/pas0001379
Valentino, K., Park, I. J. K., Cruz-Gonzalez, M., Zhen-Duan, J., Wang, L., Yip, T., €¦ Alegría, M. (2025). Family-level moderators of daily associations between discrimination and distress among Mexican-origin youth. Development and Psychopathology, 37(2), 902-917. doi: https://doi.org/10.1017/S0954579424000749
van Roekel, E., Keijsers, L., & Chung, J. M. (2019). A review of current ambulatory assessment studies in adolescent samples and practical recommendations. Journal of Research on Adolescence, 29(3), 560-577. doi: https://doi.org/10.1111/jora.12471
Vogelsmeier, L. V. D. E., Jongerling, J., & Ulitzsch, E. (2025). Accounting for measurement invariance violations in careless responding detection in intensive longitudinal data: Exploratory vs partially constrained latent markov factor analysis. Multivariate Behavioral Research, 60(5), 878-897. doi: https://doi.org/10.1080/00273171.2025.2492016
Wang, L., Hamaker, E., & Bergeman, C. (2012). Investigating inter-individual differences in short-term intra-individual variability. Psychological Methods, 17(4), 567-581. doi: https://doi.org/10.1037/a0029317
Wang, L., & Maxwell, S. E. (2015). On disaggregating between-person and within-person effects with longitudinal data using multilevel models. Psychological Methods, 20(1), 63-83. doi: https://doi.org/10.1037/met0000030
Wang, L., Zhang, Z., McArdle, J. J., & Salthouse, T. A. (2008). Investigating ceiling effects in longitudinal data analysis. Multivariate Behavioral Research, 43(3), 476-496. doi: https://doi.org/10.1080/00273170802285941
Ward, M. K., & Meade, A. W. (2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74(1), 577-596. doi: https://doi.org/10.1146/annurev-psych-040422-045007
Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063-1070. doi: https://doi.org/10.1037/0022-3514.54.6.1063
Wen, C. K. F., Schneider, S., Stone, A. A., & Spruijt-Metz, D. (2017). Compliance with mobile ecological momentary assessment protocols in children and adolescents: A systematic review and meta-analysis. Journal of Medical Internet Research, 19(4), Article e132. doi: https://doi.org/10.2196/jmir.6641
Wrzus, C., & Neubauer, A. B. (2023). Ecological momentary assessment: A meta-analysis on designs, samples, and compliance across research fields. Assessment, 30(3), 825-846. doi: https://doi.org/10.1177/10731911211067538
Yang, M., Schick, M. R., Sullivan, T. P., & Weiss, N. H. (2024). Predicting completion of ecological momentary assessments among substance-using women experiencing intimate partner violence. Assessment, 31(7), 1398-1413. doi: https://doi.org/10.1177/10731911231216948
Yip, T., Wang, Y., & Zhao, A. (2025). Promises and pitfalls of experience sampling methods for understanding developmental processes. Annual Review Developmental Psychology, 7, 439-460. doi: https://doi.org/10.1146/annurev-devpsych-111323-095203
