Can Large Language Models (LLMs) be Trusted for Power Analysis? An Empirical Evaluation

1 Introduction

Power analysis is a critical step in quantitative social and behavioral research, used to determine the minimum sample size required to detect hypothesized effects and reject null hypotheses with sufficient statistical confidence (e.g., Cohen, 1988). As a critical component of research design (Myors & Murphy, 2023), it has been widely recommended across disciplines such as psychology, education, and biomedical science (e.g., Aberson, 2019; Anderson, Kelley, & Maxwell, 2017; Finch, Cumming, & Thomason, 2001; Ioannidis, 2005; Maxwell, Lau, & Howard, 2015; Wilkinson & on Statistical Inference., 1999). Adequately powered studies yield more reliable and generalizable results, whereas underpowered studies increase the risk of false positives and reduce replicability (Ioannidis, 2005; Maxwell, 2004). Well-conducted power analysis encourages researchers to articulate clear, theoretically informed expectations about effect sizes and study design before data collection begins (Button et al., 2013; Cohen, 1988). By doing so, it helps prevent common pitfalls such as relying on samples that are too small to detect meaningful effects or drawing unwarranted conclusions from statistically non-significant results that are simply underpowered (Ioannidis, 2005; Sedlmeier & Gigerenzer, 1989). In this sense, power analysis serves not only as a technical planning tool but also as a safeguard against biased estimates and irreproducible findings that have contributed to broader concerns about research reliability across disciplines (Button et al., 2013; Fritz, Scherndl, & Kühberger, 2013).

Despite its recognized importance, power analysis remains underutilized in actual research practice. A meta-analysis of over 6,000 published studies found that only 3% reported statistical power (Fritz et al., 2013). This underuse is largely due to two interrelated barriers. The first stems from the technical and cognitive demands of conducting a valid power analysis. Researchers must understand statistical inference, including effect sizes, Type I/II errors, and model-specific parameters (Cohen, 1988; Myors & Murphy, 2023). The second relates to the accessibility and usability of analytical tools. Learning a new statistical program or programming language can be cognitively demanding (Kingsley & Robertson, 2017; Piccolo, Denny, Luxton-Reilly, Payne, & Ridge, 2023), and limited access to software or licenses can pose additional financial constraints. Power analysis is often conducted using tools such as G*Power (Faul, Erdfelder, Lang, & Buchner, 2007), the “pwr” package in R (Champely et al., 2017), SPSS (Corp., 2021), the “sampsi” and “fpower” packages in Stata (StataCorp., 2023), or SAS PROC POWER (Inc., 2023), which require either technical fluency or institutional access. Consequently, these practical barriers to performing accurate power analysis may inadvertently undermine the rigor and replicability of empirical research.

To tackle these challenges and improve the accessibility of power analysis, researchers are increasingly turning to emerging tools capable of handling statistical complexity and reducing technical barriers (Achiam et al., 2023; Frank, 2023; Wang et al., 2023). Recent studies highlight a growing reliance on tools that can support tasks requiring both domain knowledge and reasoning capabilities—areas in which LLMs have shown emerging, but still uneven, performance (T. Zhang et al., 2024). LLMs are currently already being used to assist in a variety of research tasks, including literature synthesis (Jalali & Akhavan, 2024), data interpretation (Khraisha, Put, Kappenberg, Warraitch, & Hadfield, 2024), and inferential analysis (for Social Science & Innovations, 2025).

However, their increasing use in statistical contexts raises concerns about whether such systems can provide accurate and reliable guidance, particularly for tasks such as power analysis that require precise specification of assumptions and parameters. Although LLMs are trained on statistical literature and can generate and explain procedures, it remains unclear whether these capabilities translate into methodologically sound and trustworthy outputs in practice. Meanwhile, workshops and tutorials promoting LLMs-based research design and data analysis workflows are rapidly emerging (e.g., for Social Science & Innovations, 2025; in Europe, 2025). While these initiatives often emphasize practical implementation, they appear to offer limited systematic empirical validation, particularly for tasks requiring methodological rigor. Existing studies have begun to explore these issues but are often based on narrow task scopes or specific prompting strategies (e.g., Methnani, Latiri, Dergaa, Chamari, & Saad, 2023), leaving important questions unresolved. The growing adoption of LLMs alongside limited empirical evaluation suggests the need for more systematic and critical examination. As noted by prior work, the uncritical integration of LLMs into research workflows may pose risks to established scientific standards (Birhane, Kasirzadeh, Leslie, & Wachter, 2023). Accordingly, further empirical investigation is warranted to better understand the capabilities and limitations of LLMs in statistically demanding tasks.

Given these considerations and potential methodological risks, we conducted a systematic evaluation of LLMs’ performance in power analysis. Specifically, we tested six LLMs, including ChatGPT (GPT-3.5, GPT-4, GPT-4o), Llama 3.2-3b, Claude 4.6 Sonnet, and Gemini 2.5 Flash, focusing on their ability to interpret problem descriptions and generate analytic code. Three experiments were designed to critically assess their effectiveness. Experiment 1 evaluated whether each model could correctly compute required sample sizes for common statistical tests using multiple prompting strategies. Experiment 2 examined whether the models could accurately identify and explain missing or incomplete input information necessary for valid power analyses. Experiment 3 extended Experiment 1 by evaluating model performance on more computationally complex statistical tests, specifically Pearson correlation and multiple linear regression. By examining both task performance and response consistency, this study seeks to provide an empirical evaluation of the capabilities and limitations of LLMs and to inform discussions about their appropriate use in quantitative research contexts.

1.1 Research Questions

Power analysis can serve multiple purposes, such as determining the necessary sample size, or performing post hoc analyses (Aberson, 2019). In this study, we specifically focus on sample size calculation, given its common use in research design. Thus, our first overarching research question (RQ1) addresses whether LLMs can accurately calculate sample sizes for commonly used statistical tests. To examine this question, we structured our investigation in three progressive steps.

First, recognizing that different types of LLMs vary significantly in performance and reliability, we evaluated the accuracy of sample size calculations across test types and models. The selected statistical tests (two-sample t-test, one-way ANOVA, and \(\chi ^2\) goodness-of-fit test) were selected as baseline evaluation tasks because they are widely used in applied research and involve relatively well-defined analytical procedures. To capture the variation across model families and capability levels, we examined six LLMs in this study: GPT-3.5-turbo, GPT-4, and GPT-4o (developed by OpenAI), Llama 3.2-3B (developed by Meta), Claude 4.6 Sonnet (developed by Anthropic), and Gemini 2.5 Flash (developed by Google). These models were selected to represent a range of capability levels and model ecosystems rather than to endorse any particular system. The three ChatGPT versions reflect different stages within a widely used proprietary model family, with GPT-3.5-turbo serving as a commonly used baseline, GPT-4 as a higher-capability model, and GPT-4o as the most recent optimized version available at the time of data collection. Claude 4.6 Sonnet and Gemini 2.5 Flash were included to broaden coverage across major proprietary LLM ecosystems, whereas Llama 3.2-3B was included as an open-source alternative.

Next, considering existing evidence that LLMs often struggle with direct numerical calculations (e.g., Frieder et al., 2023; Jiang et al., 2024; Mirzadeh et al., 2024) but may perform more effectively when generating code (e.g., Biswas, 2023; Liu, Xia, Wang, & Zhang, 2024; Nigar & Mohammed, 2023), we explored whether prompting LLMs to generate code rather than directly calculate numerical results would improve their accuracy (RQ1b). To explore different approaches to code generation, we included both R and Python in the experiment. By including both languages, we aimed to examine whether specifying programming language in the prompt affects the accuracy of sample size calculation tasks.

Furthermore, our second overarching research question (RQ2) addresses another potential barrier to conducting power analysis: identifying the specific statistical parameters required for performing these calculations. Effective power analysis necessitates providing a complete and precise set of inputs, yet determining these inputs is often challenging without sufficient methodological training (Cohen, 1988; Myors & Murphy, 2023). Unlike structured software tools that enumerate required fields visually (e.g., G*Power), natural-language interactions with LLMs leave the completeness of a user’s specification ambiguous, potentially exacerbating existing concerns about the misuse or misapplication of power analysis in empirical research (Fritz et al., 2013). Although prior work suggests that LLMs may be capable of flagging missing or ambiguous information in user prompts (T. C. Chen et al., 2023), it remains unclear whether they can do so accurately and consistently in the context of power analysis, where omission can fundamentally invalidate results. Thus, we examined whether LLMs can actually recognize and inform users about these critical missing elements required for valid power analysis.

Finally, although the initial experiments focused on relatively common statistical tests (two-sample t-test, one-way ANOVA, and \(\chi ^2\) goodness-of-fit test), many real-world research settings involve more computationally demanding analyses. Therefore, we further examined whether the performance patterns observed in the initial experiments would generalize to more complex statistical procedures, specifically focusing on Pearson correlation and multiple linear regression (RQ3). This extension allowed us to assess whether the performance patterns observed in the initial experiments would generalize beyond the initial set of statistical tests.

In sum, we formulated the following research questions:

Research Question 1:

Can LLMs perform sample size calculation for two-sample t-test, one-way ANOVA, and \(\chi ^2\) goodness-of-fit test?

Research Question 1a:

Do different types of LLMs affect the accuracy of sample size calculation?

Research Question 1b:

Does requesting code through LLMs yield higher accuracy than directly asking to calculate sample sizes?

Research Question 2:

Can LLMs identify missing information necessary for power analysis?

Research Question 3:

Can LLMs perform sample size calculations for Pearson correlation and multiple linear regression?

2 Experiment 1: Sample Size Calculation

2.1 Method

Experimental Design and Procedure

Experiment 1 aimed to evaluate LLMs’ accuracy in determining the required sample size. We used a 6 \(\times \) 3 \(\times \) 3 factorial experimental design, incorporating three experimental factors: “models,” “methods,” and “test types.”

First, the “model” factor included six conditions: using the GPT-3.5-turbo, GPT-4, GPT-4o (OpenAI), Llama 3.2-3b (Meta), Claude 4.6 Sonnet (Anthropic), and Gemini 2.5 Flash (Google). The specific model versions evaluated in this study were GPT-3.5-turbo (gpt-3.5-turbo-0125), GPT-4 (gpt-4-0613), GPT-4o (gpt-4o-2024-08-06), Llama 3.2-3b (Llama 3.2-3b-Instruct), Claude 4.6 Sonnet (claude-sonnet-4-6), and Gemini 2.5 Flash (gemini-2.5-flash). The initial set of models (GPT-3.5-turbo, GPT-4, GPT-4o, and Llama 3.2-3b) was evaluated in January 2025. Claude 4.6 Sonnet and Gemini 2.5 Flash were added in April 2026 as part of an expansion phase intended to incorporate newer frontier models and broaden the cross-platform comparison. Model identifiers and query periods are reported to support reproducibility and transparency. The models were selected to represent a cross-tier spectrum of LLMs, including a lightweight open-source model, widely used proprietary models, and newer frontier systems from multiple providers. This comparison was designed to test Research Question 1a.

Second, the “methods” factor had three conditions: (1) LLMs directly computing the required sample size, (2) LLMs generating R code to calculate the sample size, and (3) LLMs generating Python code for the same task. These conditions were used to test Research Question 1b.

Lastly, we considered three types of statistical tests in the “test types” factor: the two-sample t-test (Student’s t-test), the one-way ANOVA (Fisher’s F test), and the \(\chi ^2\) goodness-of-fit test (Pearson’s \(\chi ^2\) test), which are commonly used in basic statistical research design. 100 sets of parameters (e.g., significance level, effect size, power) were generated for each test type to be used as a testing sample for each mode type to ensure reliability of the estimate for statistical testing. A priori power analysis using discordant pair probabilities of 0.3 and 0.1 showed that 74 pairs were sufficient to achieve 80% power at \(\alpha \) = 0.05, supporting our choice of 100 repetitions per condition. Combining these results, we will examine the findings for Research Question 1—whether LLMs can calculate sample sizes in power analysis.

The procedures for each test type and scenario are illustrated in Figure 1. The conversational structure used in the experiment is outlined in Table 1, and a specific example of the experiment is provided in Figure 2.

Experimental Setup

In this study, we designed and implemented the experiment using the OpenAI API (for GPT models), the Anthropic API (for Claude 4.6 Sonnet), the Google AI API (for Gemini 2.5 Flash), and the Llama API (for Llama 3.2-3b). First, we developed a question list based on the conversational structure from Experiment 1. Next, we integrated each statistical test type with the relevant information into this question list, which was then input into the API. We used the API versions of these models to enable systematic and reproducible testing, allowing us to run 100 trials per condition under controlled settings. The API was used solely to standardize evaluation and does not represent a fundamentally different user workflow. In practice, researchers would more commonly interact with LLMs through conversational interfaces rather than directly through an API. However, the prompts and responses in our experiments followed a conversational format similar to publicly available chat interfaces such as ChatGPT, making the interaction structure broadly comparable to typical real-world use.

We also compiled a list of the information necessary for conducting the tests. For significance levels, we used two significance levels: \(\alpha \) = 0.01 and \(\alpha \) = 0.05. Regarding effect sizes, we arbitrarily selected a number from Cohen (1988)’s suggested criteria for small, medium, and large effect sizes, using the following set: 0.15, 0.20, 0.25, 0.28, 0.30, 0.35, 0.50, 0.60, and 0.70. Additionally, we chose three desired power levels: 0.75, 0.8, and 0.9, based on Cohen’s recommended power levels. In total, these parameter combinations yielded 100 cases, which were submitted to the API.

Depending on the test being conducted, we specified additional parameters. For the two-sample t-test, we indicated whether it was one-tailed or two-tailed. We also provided information that Group 1 and Group 2 had equal sample sizes by specifying the allocation ratio (N1=N2), focusing on this condition to examine a simplified scenario and not considering unequal group sizes. For the one-way ANOVA test, we included the number of groups as an essential variable for power analysis, using group counts of 3, 4, 6, and 10, as drawn from examples provided in Cohen (1988). In the case of the \(\chi ^2\) test, we input the degrees of freedom as necessary information, using values of 2, 4, 9, and 20, also based on Cohen (1988).

We systematically combined all these variables to create different condition sets and randomly generated 100 parameter sets within each condition. The details of the information used in the experiment are presented in Table 2.

Additionally, to minimize randomness due to the probabilistic nature of generative AI models, we set the temperature parameter to 0. The temperature is a tuning parameter in many foundational models that controls the “creativity” or randomness of generated outputs. It typically ranges from 0 to 1, with lower values leading to more deterministic and repetitive outputs, and higher values producing more diverse and creative but potentially less stable outputs. A temperature of 0 ensures maximum determinism, where the model outputs the token with the highest probability based on its training data. Our goal was to prioritize stable and replicable outcomes over creativity, which is why we chose this minimum setting.

In the first step of the experiment, we initiated each dialogue with the same prompt: “Can you help me do power analysis? I want to do [test type].” This provided context for LLMs to guide users in gathering the necessary information for conducting power analysis for the specified test. Once this context was established, the second step of the experiment varied based on the scenario (see Figure 1). We input different combinations of information from the compiled list in subsequent prompts, asking LLMs to either directly calculate the sample size or generate the required R or Python code to do so. With the temperature set to 0, the outputs remained stable and consistent in most cases. Based on the given context, LLMs generated appropriate responses to the provided information.

At the end of the second conversation, we added several sentences to facilitate prompt engineering and improve accuracy. For example, to minimize potential randomness, we used phrases like “Use the existing code as it is” and “Do not round up.” This was because LLMs occasionally round numbers or extracts only the most convenient information for users, which can introduce unnecessary errors. Our aim was to ensure consistent use of basic code. Additionally, commands like “Take a deep breath. Do not make a mistake.” leveraged the model’s zero-shot capabilities (Kojima, Gu, Reid, Matsuo, & Iwasawa, 2023) to further enhance accuracy. These strategies were applied throughout the experiment whenever we asked LLMs to calculate or generate code. Overall, the prompts were kept simple and consistent to reflect typical researcher-LLM interactions rather than relying on heavily engineered prompts, allowing us to evaluate model performance under realistic usage conditions and attribute observed differences to the models themselves rather than to prompt variation.

Analyses

To evaluate performance, we defined accuracy as the number of times the required sample size was correctly calculated, divided by the total number of trials. When LLMs were asked to generate R or Python code, we executed the code to verify that it ran successfully and produced the correct output. We also calculated the standard error and confidence intervals to provide uncertainty estimates for the accuracy due to the limited number of replications.

Our analysis focused on the effects of the manipulations, including models, test types, and sample size calculation methods, on accuracy. To compare our experimental conditions, we applied pairwise McNemar tests to evaluate the differences between models, test types, and sample size calculation methods. Specifically, we compared (a) LLMs (GPT-3.5-turbo, GPT-4, GPT-4o, Llama 3.2-3b, Claude 4.6 Sonnet, and Gemini 2.5 Flash) while holding the statistical test type and sample size calculation approach constant; (b) statistical test types (t-test, one-way ANOVA, and chi-square test) while holding the LLM and calculation approach constant; and (c) sample size calculation approach (direct LLM-generated calculations, R-based code, and Python-based code) while holding the LLM and test type constant. Since our outcome was binary (coded as 1 for success and 0 for failure), we used the nonparametric pairwise McNemar test to determine whether the differences between pairs were statistically significant. We used the ‘rcompanion’ library in R for the pairwise McNemar test. To control for multiple comparisons, we adjusted the p-values using the Benjamini–Hochberg procedure for controlling the False Discovery Rate (FDR), ensuring that the overall rate of false discoveries remained below a specified threshold. The threshold for FDR-corrected significance, above which variables are rejected, is set to 0.05 (Benjamini & Hochberg, 1995).

2.2 Results

In Experiment 1, the results indicate that the GPT-4, GPT-4o, and Claude 4.6 Sonnet models perform best when generating R code to compute the required sample size with an accuracy of 1.00 (100.00%). The results of Experiment 1 are presented in Table 3 and Figure 3.

To answer Research Question 1, GPT-4, GPT-4o, and Claude 4.6 Sonnet achieved perfect accuracy when generating R code for sample size calculation, demonstrating that LLMs can successfully perform power analysis when appropriate coding strategies are used. GPT-3.5-turbo also showed strong performance, whereas Gemini 2.5 Flash exhibited occasional failures in ANOVA and \(\chi ^2\) scenarios. Overall, contemporary proprietary LLMs were generally capable of generating functional code for statistical power analysis, although performance varied across statistical tests and programming environments.

Python-based code generation was also generally successful but yielded lower performance than R, particularly for the \(\chi ^2\) goodness-of-fit test. One contributing factor is that Python and R implement \(\chi ^2\) power analysis differently. In Python, the relevant function requires the number of categories (n_bins), whereas R and G*Power use degrees of freedom as input. Models frequently supplied degrees of freedom to Python functions, leading to execution failures or incorrect results. In contrast, direct calculation was largely unsuccessful across models, consistent with previous findings (e.g., Methnani et al., 2023). Rather than producing correct sample sizes, models often generated incorrect numerical answers or provided procedural instructions for conducting the analysis in software such as G*Power.

When models failed, their errors generally fell into three categories. The first was function hallucination, in which models invoked functions that do not exist in the relevant package or selected an inappropriate function for the requested analysis. The second was wrong input/output specification, in which models correctly identified the relevant function but supplied unsupported arguments or incorrectly handled the function’s inputs or outputs. The third was wrong calculation, in which models produced an incorrect numerical result despite identifying the appropriate analytical procedure. Unlike function hallucination or input/output errors, wrong-calculation errors often generated plausible numerical answers with no obvious indication that the result was incorrect, making them particularly difficult for users to detect. Representative examples of each error type are provided in Appendix A.

Pairwise McNemar Test

We conducted pairwise comparisons for each condition to evaluate our research questions. Because performance was operationalized as a binary outcome (coded as 1 for success and 0 for failure), we used pairwise McNemar tests to compare between conditions. These tests were applied by pairing outcomes within the same condition while holding the remaining factors constant, which allows us to assess whether success rates differed significantly between pairs of models, test types, or calculation approaches.

For the model comparison, GPT-4, GPT-4o and Claude Sonnet 4.6 performed equivalently (p = .337, .449, and .833 respectively). Three models significantly outperformed GPT-3.5-turbo and Gemini 2.5 Flash (p < .001). GPT-3.5-turbo and Gemini 2.5 Flash did not differ significantly from each other (p = .118), suggesting similar intermediate-level performance. These findings indicate that model type substantially influenced sample size calculation accuracy, supporting Research Question 1a.

For the method comparison, instructing GPT to generate R code was more effective than generating Python code (p < .001), and generating Python code was more effective than direct calculation (p < .001). These results support Research Question 1b by demonstrating that code-generation prompting strategies yield higher accuracy than direct numerical calculation.

For the test-type comparison, all pairwise comparisons were significant. The two-sample t-test yielded higher accuracy than both one-way ANOVA (p < .001) and the \(\chi ^2\) test (p < .001), and ANOVA also outperformed the \(\chi ^2\) test (p < .001), indicating that LLM performance varied systematically across statistical test types.

The pairwise McNemar test was used to compare the results of each paired experimental condition for paired categorical groups, given the binomial nature of the outcomes. The results of the pairwise McNemar tests are shown in Table 4. In Table 6, we report both original (unadjusted) p-values and the FDR-adjusted p-values, using the Benjamini–Hochberg procedure (Benjamini & Hochberg, 1995). Presenting both values enhances transparency and interpretability, helping to assess the impact of the adjustment on the significance of the results.

3 Experiment 2: Identifying Missing Information

3.1 Method

Experimental Design and Procedure

In Experiment 2, our goal was to examine whether LLMs could identify missing information required for power analysis and assist users in completing the process. We deliberately omitted one piece of information and assessed whether LLMs could recognize it, then calculated its corresponding accuracy. The experiment followed the same model settings (GPT-3.5-turbo, GPT-4, GPT-4o, and Llama 3.2-3b) and data collection procedures as Experiment 1, with Claude 4.6 Sonnet and Gemini 2.5 Flash added during a later expansion phase in April 2026.

As in Experiment 1, Experiment 2 used the same statistical tests and input information: the two-sample t-test, one-way ANOVA, and \(\chi ^2\) goodness-of-fit test, as detailed in Table 2. The first part of the dialogue followed the same structure as in Experiment 1. In the second part, we sequentially omitted one piece of information from Table 2 (such as significance level, effect size, power, or another key parameter) and asked LLMs if there was missing information. Each scenario, with one missing piece of information, was tested 100 times. The conversational structure is outlined in Table 5, and the procedures, along with examples of inputs used in the prompts, are shown in Figures 4 and 5.

Note. Step 1 was identical to Experiment 1.

Experimental Setup

We used the same experimental setup as in Experiment 1. This experiment was conducted using the OpenAI API (for GPT models), the Anthropic API (for Claude 4.6 Sonnet), the Google AI API (for Gemini 2.5 Flash), and the Llama API (for Llama 3.2-3b). First, we created a question list based on the conversational structure from Experiment 2. Next, we compiled the list by pairing each statistical test type with its corresponding information, omitting one piece of data. The compiled question list was then input into the API. To perform the experiment, we set the temperature to 0, applied prompt engineering techniques, and used the initial dialogue as the default, consistent with the procedures in Experiment 1.

Analyses

To assess performance, we calculated accuracy by evaluating the correctness of results after conducting 100 trials for each condition. Accuracy was defined as the proportion of correctly identified missing components or pieces of information required for conducting a valid sample size calculation, and it was calculated as the number of correct identifications divided by the total number of trials. Additionally, we computed the standard error and confidence intervals for the accuracy estimates, taking into account the limited number of replications. For Experiment 2, the focus was on parameter-specific accuracy, for which descriptive analysis with uncertainty estimates was deemed sufficient without testing differences.

3.2 Results

To address Research Question 2, we examined whether LLMs could identify missing information required for valid power analyses. Especially, when significance level, effect size, and power—key components for conducting power analysis—were omitted, GPT-4o achieved the highest accuracy of 1.00 (100%), followed by GPT-3.5-turbo at 0.99 (98.6%), Gemini 2.5 Flash at 0.97 (97.3%), Claude 3.7 Sonnet at 0.97 (96.6%), and GPT-4 at 0.96 (95.9%). Llama showed substantially lower accuracy (51.78%) across most conditions. The results of Experiment 2 are presented in Table 6 and Figure 6.

Performance varied depending on the parameter being omitted. Missing significance levels, effect sizes, desired power values, and test-specific parameters (e.g., number of groups or degrees of freedom) were generally identified by LLMs. In contrast, missing tail specification for the two-sample t-test proved particularly challenging. Claude 3.7 Sonnet correctly identified the missing tail type in all cases, whereas other models frequently failed to recognize the omission and instead implicitly assumed a two-tailed test. Similarly, missing allocation ratios were detected reliably by Claude 3.7 Sonnet, GPT-4, and GPT-4o but less consistently by Gemini 2.5 Flash and Llama.

When models failed to identify missing information, they often proceeded as though all required inputs had been provided. Common responses included statements such as “There is no missing information” or direct attempts to perform the calculation without first verifying parameter completeness. In some cases, models substituted default assumptions rather than explicitly requesting the missing information. For example, omitted significance levels were often assumed to be .05, desired power was frequently assumed to be .80, and missing tail specifications were commonly treated as two-tailed tests. In the \(\chi ^2\) condition, some models incorrectly inferred that expected frequencies were missing when the actual missing parameter was desired power. Representative examples of model responses that failed to identify missing parameters are provided in Appendix B.

These findings suggest that contemporary LLMs are generally capable of identifying missing information in common power-analysis scenarios, particularly for standard parameters such as significance level, effect size, and desired power. However, performance declines when omitted parameters are less salient or more context dependent, such as tail specification or allocation ratio. Consequently, users should not assume that LLMs will reliably detect all missing inputs and may benefit from explicitly asking models to verify parameter completeness before conducting power analyses.

4 Experiment 3: Sample Size Calculation for Pearson Correlation and Multiple Regression

4.1 Method

Experimental Design and Procedure

Experiment 3 extended Experiment 1 to examine whether the performance patterns observed for the two sample t-test, ANOVA, and \(\chi ^2\) test generalize to two statistically more complex tests common in behavioral and social research: Pearson correlation and multiple linear regression. We used a 4 \(\times \) 3 \(\times \) 2 factorial experimental design, incorporating three experimental factors: “models,” “methods,” and “test types.”

First, the “model” factor included four conditions: Claude 4.6 Sonnet (Anthropic), GPT-4o (OpenAI), Gemini 2.5 Flash (Google), and Llama 3.2-3b (Meta). The specific models selected represent the current state of frontier LLMs, extending Experiment 1 with newer model versions across multiple providers.

Second, the “methods” factor had three conditions: (1) LLMs directly computing the required sample size, (2) LLMs generating R code to compute the required sample size, and (3) LLMs generating Python code to compute the required sample size. These conditions are identical to those used in Experiment 1.

Lastly, we considered two types of statistical tests in the “test types” factor: Pearson correlation and multiple linear regression. These tests were selected because they involve more complex computations than the two sample t-test, ANOVA, and \(\chi ^2\) tests in Experiment 1: Pearson correlation uses the Fisher z transformation, while multiple regression requires solving the non-central F distribution numerically. For each test type, we randomly generated 100 parameter sets per condition, yielding 4 \(\times \) 3 \(\times \) 2 \(\times \) 100 = 2,400 total trials.

Experimental Setup

We used the same experimental setup as in Experiment 1, with the following updates. We examined a range of parameter combinations across three types of analyses. For Pearson correlation, the correlation coefficient (r) was varied across 0.10, 0.20, 0.30, 0.50, and 0.70, with significance levels (\(\alpha \)) of 0.01 and 0.05 and target power levels of 0.75, 0.80, and 0.90. Ground-truth sample sizes were computed using the Fisher z closed-form formula (Cohen, 1988), which is mathematically equivalent to the procedure implemented in the R function pwr::pwr.r.test(). For multiple linear regression, the coefficient of determination (\(R^{2}\)) includes 0.02, 0.05, 0.10, 0.15, 0.25, and 0.35, and the number of predictors was set to 1, 2, 3, or 5, again with \(\alpha \) values of 0.01 and 0.05 and target power levels of 0.75, 0.80, and 0.90. Ground-truth sample sizes were computed in R by numerically solving the non-central F distribution, following the same procedure implemented in pwr::pwr.f2.test() (Cohen, 1988). The two-turn conversation structure and three prompt conditions from Experiment 1 were retained.

Analyses

We used the same analytical approach as in Experiment 1. To evaluate performance, we defined accuracy as the number of times the required sample size was correctly calculated, divided by the total number of trials.

4.2 Results

To answer Research Question 3, by instructing Claude 4.6 Sonnet and Gemini 2.5 Flash to generate R code, they both demonstrated perfect accuracy (1.00) for both Pearson correlation and multiple regression. However, overall performance across models was somewhat less consistent than in the simpler statistical scenarios examined earlier, such as the two sample t-test, one-way ANOVA, and \(\chi ^2\) test. These findings suggest that although contemporary LLMs can also generate code for more advanced analyses, performance may become less stable as statistical procedures and parameter specifications increase in complexity. The results of Experiment 3 are presented in Table 7 and Figure 7.

To address Research Question 3, we examined whether the performance patterns observed in Experiment 1 would extend to Pearson correlation and multiple linear regression. Consistent with Experiment 1, R-code generation yielded the highest overall accuracy (.62), substantially outperforming both Python-code generation (.24) and direct calculation (.21). Overall, Claude 4.6 Sonnet and Gemini 2.5 Flash showed the strongest performance in Experiment 3, whereas Llama 3.1 8B performed poorly across conditions. These findings suggest that the advantage of code generation observed in Experiment 1 extends to Pearson correlation and multiple regression, although performance varied more across models and statistical procedures.

Performance differed substantially between the two tests. Claude 4.6 Sonnet and Gemini 2.5 Flash performed well on Pearson correlation across code-generation conditions, but accuracy declined markedly for multiple regression outside the R environment. In particular, Python-based multiple regression proved challenging because the analysis requires numerical optimization rather than a straightforward library call. As a result, models that performed well on the statistical tests examined in Experiment 1 did not always maintain the same level of performance for these more advanced analyses.

When errors occurred, they largely reflected the similar failure patterns identified in Experiment 1, including function hallucinations, incorrect handling of function inputs or outputs, and wrong formula. Notably, wrong-formula errors were more evident in Experiment 3, as models occasionally applied inappropriate statistical procedures while still producing plausible numerical results, making them difficult to detect at first glance. As a result, users should verify not only the final sample size estimate but also the statistical procedure used to obtain it. Representative examples are provided in Appendix C.

5 Discussion

While LLMs are increasingly integrated into research workflows, their ability to support statistically sophisticated procedures such as power analysis remains underexplored, raising concerns about unexamined adoption. To address this gap, we systematically evaluated the performance of six LLMs, including ChatGPT (GPT-3.5-turbo, GPT-4, and GPT-4o), Llama 3.2-3b, Claude 4.6 Sonnet, and Gemini 2.5 Flash, across three experiments. Experiment 1 assessed their ability to calculate sample sizes for common statistical tests under different prompting strategies, and Experiment 2 examined whether they could identify missing information necessary for valid power analysis. Experiment 3 extended Experiment 1 to assess whether these performance patterns generalize to Pearson correlation and multiple linear regression.

5.1 Sample Size Calculation Tasks

Performance across Different Models

Across Experiments 1 and 3, substantial differences emerged across LLMs in their ability to perform power analysis tasks. Under optimal conditions, GPT-4, GPT-4o, and Claude 4.6 Sonnet achieved near-perfect or perfect performance, whereas GPT-3.5-turbo and Gemini 2.5 Flash generally showed intermediate performance and Llama consistently underperformed. These findings suggest that the ability to support statistical tasks varies considerably across models and should not be assumed to be a general characteristic of all LLMs. This finding extends previous work by Methnani et al. (2023), who observed failures in direct sample size calculations by ChatGPT, by employing common designs, conducting 100 replications per condition, and systematically favoring a code retrieval strategy over direct calculations.

At the same time, model performance was not entirely stable across statistical procedures. Several performance patterns observed in Experiment 1 were replicated in Experiment 3, but the relative ranking of models changed across tests. Models that performed strongly on common procedures such as t-tests, ANOVA, and \(\chi ^2\) tests did not always maintain the same advantage when applied to Pearson correlation and multiple regression. This finding suggests that successful performance in statistical tasks depends not only on overall model capability but also on the specific computational and methodological requirements of a given procedure.

These results have important implications for evaluating LLMs in research settings. Performance on a limited set of benchmark tasks may provide an incomplete picture of a model’s statistical capabilities. A model that performs accurately on familiar and well-defined analyses may still struggle when faced with procedures that require different computational approaches or less common statistical implementations. More broadly, the findings suggest that LLM capability for power analysis is better understood as domain-specific rather than global. Researchers should therefore avoid assuming that strong performance on one type of statistical task will necessarily generalize to others.

Effectiveness of Prompting Strategies

Beyond differences between models, the way tasks were presented to LLMs had a substantial impact on performance. Across Experiments 1 and 3, prompting models to generate code consistently produced more accurate results than asking them to perform calculations directly. This finding aligns with prior research suggesting that contemporary LLMs are generally more effective at generating executable code than performing precise mathematical computations internally (e.g., Biswas, 2023; Liu et al., 2024; Nigar & Mohammed, 2023). Rather than directly performing statistical calculations, LLMs appear to be more effective at translating statistical problems into executable code that can be evaluated by software.

Within the code-generation conditions, R consistently outperformed Python. One possible explanation is that statistical power analysis is more commonly implemented and documented in R-based research workflows, providing models with greater exposure to relevant examples during training. Another possibility is that some Python implementations require additional programming steps or software-specific knowledge, creating more opportunities for errors. For example, several failures stemmed from incorrect handling of package-specific arguments or outputs rather than from misunderstandings of the underlying statistical concepts.

Importantly, code generation should not be equated with correctness. Although many generated solutions were accurate, models occasionally produced function hallucinations and wrong-output-field errors. Function hallucinations occurred when models invoked functions that did not exist in the relevant package or selected inappropriate functions for the requested analysis. Wrong-output-field errors occurred when models correctly identified a statistical function but attempted to extract information that was not returned by the function. These failures highlight that code can appear plausible while still being nonfunctional.

In addition, Experiment 3 revealed examples of wrong-formula errors, in which models applied an inappropriate statistical procedure despite generating executable code. Unlike function hallucinations or wrong-output-field errors, wrong-formula errors often produced plausible numerical outputs without triggering runtime failures, making them particularly difficult for users to detect. Consequently, evaluating only whether code executes successfully may provide a false sense of correctness. Researchers should therefore verify not only the functionality of generated code but also the appropriateness of the underlying statistical procedure.

These findings suggest that performance in power analysis depends not only on the capabilities of a particular model but also on how users structure their interactions with it. Careful prompting can substantially improve performance, yet even high-performing models remain susceptible to subtle coding and methodological errors. As a result, LLM-generated code should be treated as a draft that requires independent verification rather than as a final analytical product.

5.2 Identification Tasks for Missing Information

Experiment 2 explored whether LLMs could identify missing input parameters required for conducting valid power analysis. Overall, performance was strongest when identifying commonly required parameters such as significance levels, effect sizes, and desired power values. This finding suggests that LLMs possess a degree of procedural knowledge about the structure of power analysis and can often recognize when essential inputs are absent.

However, performance was less consistent for test-specific parameters. Models frequently identified broadly applicable inputs while overlooking details that depend on the particular statistical design, such as tail specification or allocation ratios. This pattern suggests that LLMs may not always distinguish between information that is explicitly required and information that can be reasonably assumed. Instead, they often appear to rely on conventions commonly encountered in training data, such as assuming a two-tailed test or a default power level of .80, rather than prompting users to provide the missing information. While these assumptions may sometimes be reasonable, they are not necessarily appropriate for a given research context.

This finding highlights a broader limitation of LLM-assisted statistical reasoning. Identifying missing information is not simply a matter of recalling which parameters are associated with a statistical procedure; it also requires recognizing when those parameters remain underspecified in a particular research design. In this sense, the challenge is not simply identifying which parameters are relevant to a statistical test, but determining whether those parameters have been adequately specified in a particular analysis. The tendency of LLMs to proceed with implicit assumptions suggests that they may prioritize producing a complete response over explicitly acknowledging uncertainty or incompleteness. As a result, users may receive outputs that appear methodologically sound while resting on assumptions that were never verified.

From a practical perspective, these findings suggest that LLMs may be more useful as aids for structuring power analyses than as autonomous methodological advisors. Rather than assuming that a model will automatically identify all missing inputs, researchers should actively prompt the model to verify parameter completeness and explicitly state any assumptions used in the analysis. Such verification is particularly important for test-specific design choices, where omissions may have meaningful consequences for sample size estimation. To facilitate this process, Figure 8 provides a recommended verification workflow corresponding to the statistical tests examined in this study.

5.3 Limitations and Future Direction

This study has several limitations, each of which provides insights into areas where future research can build upon. First, the evaluation focused primarily on relatively structured power-analysis scenarios involving two-sample t-tests, one-way ANOVA, \(\chi ^2\) goodness-of-fit tests, Pearson correlation, and multiple linear regression. These procedures were intentionally selected because they have well-established analytical solutions, allowing objective verification of correctness and providing a necessary baseline for evaluating LLM performance. However, many real-world power analyses involve substantially greater complexity, including multilevel models, structural equation models, repeated-measures designs, and simulation-based approaches. Although Experiment 3 extended the evaluation to Pearson correlation and multiple regression, these analyses still rely on established statistical procedures and software implementations.

An important question raised by the present findings is whether LLM limitations stem primarily from statistical complexity itself or from difficulties implementing statistical procedures in software. In several cases, models appeared to understand the underlying statistical task but failed because of software-specific issues, such as incorrect function calls or inappropriate handling of function arguments and outputs. This distinction may be particularly important for simulation-based power analyses, which often require researchers to develop custom computational workflows rather than rely on established power-analysis functions. Evaluating LLM performance in these settings could help determine whether current models are primarily retrieving familiar coding patterns or demonstrating broader statistical reasoning capabilities. Future research should therefore examine LLM performance on more complex power-analysis scenarios, particularly simulation-based approaches that are increasingly common in applied research (e.g., Arend & Schäfer, 2019; Arnold, Hogan, Colford, & Hubbard, 2011; Moshagen & Bader, 2023; Qin, 2023; Taylor & Bosch, 1990; Z. Zhang, 2014).

Second, the study evaluated specific model snapshots available at the time of data collection, including GPT-3.5-turbo (gpt-3.5-turbo-0125), GPT-4 (gpt-4-0613), GPT-4o (gpt-4o-2024-08-06), Llama 3.2-3B (Llama 3.2-3B-Instruct), Claude 4.6 Sonnet (claude-3-7-sonnet-20250219), and Gemini 2.5 Flash (gemini-2.5-flash-preview-04-17). Because LLMs are updated frequently, the findings should be interpreted as evaluations of these specific model versions rather than stable estimates of future performance. More broadly, rapid model evolution presents a methodological challenge for LLM benchmarking research, as performance estimates may become outdated more quickly than traditional software evaluations. Future studies would benefit from maintaining reproducible benchmark tasks and reporting exact model identifiers to facilitate longitudinal comparisons across model generations.

Third, this study evaluated LLMs primarily through direct, one-shot interactions. Although this approach allowed controlled comparisons across conditions, it does not fully capture how researchers may use LLMs in practice. Real-world use often involves iterative clarification, error checking, and refinement of prompts. The present findings suggest that performance is influenced not only by the capabilities of a model but also by how users frame tasks and respond to model outputs. Statistical performance should be viewed as a product of both model capability and user interaction rather than as a fixed property of the model alone. Future research should investigate not only whether LLMs make statistical errors, but also whether users can successfully identify and correct those errors during real-world analytical tasks.

Fourth, the present study focused on the accuracy of LLM outputs rather than their practical value in real-world research workflows. This focus was intentional, as systematic evaluation of accuracy was necessary to establish a baseline assessment of LLM performance in power analysis. However, accuracy alone does not capture whether LLMs provide meaningful benefits to researchers in practice. Established tools such as G*Power and dedicated R packages already offer highly reliable sample size calculations once the appropriate parameters have been specified. Consequently, the value of LLMs may lie less in performing calculations themselves and more in assisting users with tasks such as identifying required inputs, clarifying methodological assumptions, and interpreting statistical procedures. Future research should therefore compare LLM-assisted and traditional workflows in terms of usability, efficiency, user understanding, and error rates to determine the practical contexts in which LLMs provide added value beyond existing statistical software.

6 Conclusion

Given the increasing integration of LLM-based approaches into research workflows, it is important to examine how reliably these models perform in statistically demanding contexts. Therefore, this study aims to systematically evaluate the performance of the selected LLMs in the context of power analysis rather than assume their usefulness, with the broader goal of promoting more rigorous and responsible practices of AI in research.

Overall, the findings indicate that the evaluated model versions demonstrate promising yet uneven capability in supporting sample size calculations for power analysis. On the one hand, GPT-4, GPT-4o, and Claude 4.6 Sonnet frequently retrieved appropriate statistical formulas, offered structured guidance, and generated executable code for sample size estimation tasks. Experiment 3 further demonstrated that this advantage for R-code generation extends to Pearson correlation and multiple linear regression, although performance became more variable across models and statistical procedures. These outcomes suggest that the tested models can appear competent and may assist users in navigating certain aspects of power analysis under controlled conditions.

However, consistent with prior studies (e.g., Frieder et al., 2023), the results also confirmed notable limitations in the evaluated LLMs’ direct mathematical reasoning abilities. When tasked with performing numerical calculations, these model versions often produced incorrect values due to rounding errors, inappropriate substitutions, or incomplete execution of formulas. These issues likely stem from the models’ token-based probabilistic pattern matching rather than genuine mathematical understanding (Jiang et al., 2024; Mirzadeh et al., 2024). Moreover, the accuracy of the tested models varies across different statistical tests, and in some cases, the model misinterprets missing data or default to assumptions that require careful verification, which may affect the validity of the results if left unchecked.

Given that inaccuracies in sample size calculation may affect study design, these findings highlight the importance of careful validation when using LLM-generated outputs. LLMs may serve as useful aids in structuring or supporting aspects of the analytical process, but they should not be treated as substitutes for domain expertise. Outputs should be verified using established methods, and interpretation should involve appropriate methodological oversight (van Dis, Bollen, Zuidema, van Rooij, & Bockting, 2023; Zhu, Jiang, Yang, & Ren, 2023).

Beyond computational considerations, broader issues of research accountability and ethical use must also be addressed. As LLMs become increasingly embedded in research workflows (e.g., Aubin Le Quéré et al., 2024; Khraisha et al., 2024; Methnani et al., 2023; Watkins, 2024), issues of research accountability and ethical use grow more critical. Although models such as GPT-4 and GPT-4o demonstrated strong performance in this study, none of the evaluated model versions consistently produced flawless results, underscoring the ongoing necessity for human oversight. Furthermore, concerns regarding data security and confidentiality must be taken seriously; researchers should refrain from inputting sensitive or identifiable information into LLMs, particularly when working with contextual or study-specific data (Charfeddine, Kammoun, Hamdaoui, & Guizani, 2024). Ultimately, while LLMs might offer valuable potential as supplemental tools for statistical analysis, their responsible integration requires critical oversight, human expertise, rigorous cross-validation, and a continued commitment to preserving the standards of scientific rigor. Further research is needed to better understand their capabilities and limitations across a wider range of analytical contexts.

Author Notes

Correspondence concerning this article should be addressed to Feng Ji, Department of Applied Psychology and Human Development, University of Toronto, 252 Bloor West Street, Toronto, Ontario, Canada. Email: f.ji@utoronto.ca

Funding

This study was funded by the Connaught Fund (#520245) and Social Sciences and Humanities Research Council (#519992), International Network of Educational Institutes Seed Funding (# 522011) and CRC (#001169).

Conflicts of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Availability of data and materials

All data are available at https://osf.io/m9ztc.

Code availability

All code is available at https://osf.io/m9ztc.

Open Practices Statement

The code, experiments, and results used to support the findings of this study are available at https://osf.io/m9ztc under the Creative Commons Attribution NonCommercial-ShareAlike 4.0 International license (CC BY-NC-SA 4.0). None of the experiments were preregistered.

References

library(pwr) 
alpha <- 0.05; effect_size <- 0.5; desired_power <- 0.8 
result <- power.t.test(n = NULL, delta = effect_size, sig.level = alpha, power = desired_power, type = "two.sample", alternative = "two.sided") 
print(ceiling(result$n)) 

from statsmodels.stats.power import GofChisquarePower 
effect_size = 0.1; alpha = 0.01; power = 0.8; df = 20 
analysis = GofChisquarePower() 
result = analysis.solve_power(effect_size=effect_size, nobs=None, alpha=alpha, power=power, df=df) 
print(’Required␣sample␣size:␣%.3f’ % result)

Based on the input parameters, the required sample size for each group is approximately 1,036 participants

library(pwr) n <- pwr.r.cor.test(r = 0.1, power = 0.8, sig.level = 0.05, alternative = "two.sided") 
print(paste("Required␣sample␣size:", n))

library(pwr) 
R2 <- 0.02 
f2 <- R2 / (1 - R2) 
u <- 1 
alpha <- 0.05 
power <- 0.8 
result <- pwr.f2.test(u = u, f2 = f2, sig.level = alpha, power = power) 
cat("Required␣sample␣size:", ceiling(result), "\n")

import numpy as np 
from statsmodels.stats.power 
import NormalIndPower 
r = 0.1; alpha = 0.05; power = 0.8 
effect_size = np.arctanh(r) 
power_analysis = NormalIndPower() 
sample_size = power_analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power, alternative=’two-sided’) 
print(f"Required␣sample␣size:␣{sample_size}")