1 Introduction

One goal of psychometric scales is to classify individuals, though the purpose of these classifications varies across both field and scale. In clinical psychology, diagnostic scales (e.g., the ASP; Brown, Tollefson, Dunn, Cromwell, & Filion 2001; the DMQ; Rosenthal et al. 2021) can be used to classify individuals as having, or not having, a clinically relevant level of a specific disorder. The disorder is represented as a latent trait that relates to a gold standard defined by given diagnostic criteria (e.g., DSM-5TR; American Psychiatric Association,  2022). Direct assessment of the gold standard usually requires testing and documentation of symptoms by a licensed clinician, which can be costly, time-intensive, and potentially invasive. For these reasons, the use of a gold standard may be infeasible in some contexts, and researchers may instead administer a scale to approximate the gold standard through a set of questions or items (Gibbons et al.,  2013; Gonzalez,  2021).

After obtaining a participant’s item responses, researchers interested in making a binary diagnostic classification (e.g., diagnosed vs. not diagnosed) often take one of two approaches to create these classifications. The first requires the researcher to use a parametric approach to determine a single continuous score estimating the individual’s level of the latent trait and then determine a cut-point such that only individuals with scores falling above the cut-point are placed in the diagnosed class. The second approach utilizes a non-parametric algorithm to predict the probability of diagnosis directly from the item responses and then classifies respondents based on whether the probability of diagnosis is above a given cut-point. This paper focuses on the use of item response theory (IRT) as the parametric approach and the use of random forest (RF; Breiman,  2001) with a Bayes classifier as the nonparametric approach.

Although both IRT and RF have been shown to perform well in diagnostic classification contexts, notably less research exists examining the use of RF (e.g., Giannouli & Kampakis,  2024; Ohiri, Momoh, Christopher, Ikeanumba, & Benedict,  2024) than IRT (e.g., Carvalho, Costa, Otoni, & Junqueira,  2019; R. Liu, Huggins-Manley, & Bulut,  2018; Rosenthal et al.,  2021), with even less research comparing IRT-based classification and RF-based classification directly. One previous study found that IRT-based scoring outperformed nonparametric machine learning (ML) approaches (Gonzalez,  2021). To our knowledge, no studies have examined how differential item functioning (DIF) affects this comparison.

One key challenge for classification performance is DIF, which occurs when participants with the same level of a latent trait respond differently to a given item. Many published diagnostic scales contain DIF (e.g., Golay, Abrahamyan Empson, Mebdouhi, Conus, & Alameda,  2023; Spann, Cicero, Straub, Pellegrini, & Kerns,  2024; Wardell, Cunningham, Quilty, Carter, & Hendershot,  2020); therefore, the assumption that scales are free of DIF is unlikely to hold in practice. For example, previous research suggests that individuals with autism respond differently to misophonia questionnaires than those without autism (Williams, Cascio, & Woynaroski,  2022). Misophonia is a condition characterized by a decreased tolerance for specific auditory stimuli (i.e., triggers), followed by strong, negative emotional, physiological, and behavioral responses, which often co-occurs with other neuropsychiatric conditions like autism (Schroder, Vulink, & Denys,  2013). One study illustrated that those with autism more readily indicate that their sound sensitivities interfere with their social life than those without (Williams et al.,  2022). Score differences on these items, therefore, represent co-morbidity-based differences in the pattern of responding rather than true differences in misophonia levels. This over-endorsement leads to individuals with autism having greater observed scores for misophonia than those without, even when their true levels of misophonia are identical. In this case, these items exhibit DIF across co-morbidity groups (Williams et al.,  2022). The presence of DIF can decrease classification performance, making it critical to evaluate how robust different classification methods are when DIF exists but cannot be detected.

If DIF is left unaddressed, IRT scores can be biased (Paulsen, Svetina, Feng, & Valdivia,  2020), making IRT-based classifications less accurate. Traditional DIF detection methods typically involve researchers predefining the groups across which DIF is thought to occur (Battauz,  2019; Woods, Cai, & Wang,  2013). In cases where DIF manifests in complex subpopulations (e.g., LatinX women whose highest degree obtained is a high school diploma) or unmeasured factors, like a participant’s primary language, this approach is problematic.

When DIF is suspected but groups are unknown, mixture modeling approaches like mixture IRT and factor mixture modeling offer psychometric alternatives that explicitly model latent group structure. However, these methods require researchers to specify the number of classes and the structure of classes a priori. Additionally, mixture IRT can face identification challenges when groups have identical trait distributions but differ only in item parameters (i.e., DIF without impact), particularly when DIF affects relatively few items or is moderate in magnitude (Sen & Cohen,  2024). In our study, many conditions involve no impact (identical generating trait distributions across groups), which may make mixture IRT implementations particularly challenging. However, RF can flexibly model high-dimensional interactions between items, potentially making classifications more robust to unmodeled DIF. As such, we hypothesize that RF will maintain classification performance in the presence of unmodeled DIF by capturing complex non-linear patterns in the data, making it a promising alternative for classification with unmodeled DIF (Breiman,  2001). This study focuses on the comparison between single-group IRT-based classification (without group-specific parameters) and RF-based classification, representing a common applied scenario where DIF may be left unmodeled (e.g., the grouping variable is either unmeasured, unknown, or too complex to model). We use single-group IRT as a baseline representing psychometric practice when researchers suspect DIF exists but lack information about potential grouping variables and therefore likely leave DIF unmodeled. For simplicity, we use IRT throughout the remainder of the manuscript to refer to single-group IRT.

The goals of this study were to (1) evaluate how classification accuracy changes when DIF is present but unmodeled in IRT-based classification approaches, (2) evaluate whether RF-based classification is more robust to unmodeled DIF than IRT, and (3) analyze the impact of various data characteristics on the performance of IRT and RF methods. The remainder of this paper is structured as follows. First, we outline the goals of diagnostic assessment. Then, we describe IRT- and RF-based classification methods. Next, the details of the Monte Carlo simulation study are provided, and results are discussed. This is followed by an empirical example illustrating the use of both IRT and RF on data related to misophonia. Finally, we conclude with a discussion of our findings and practical recommendations regarding the strengths and limitations of IRT and RF approaches to classification.

1.1 Diagnostic Assessment

To better understand the process of diagnostic assessment, consider diagnosing misophonia using the Duke Misophonia Questionnaire (DMQ; Rosenthal et al.,  2021). The DMQ generates a score (\(\theta _{Assessment}\)) reflecting the severity of misophonia. A threshold (\(T\)) is then applied to classify individuals as having misophonia (\(D_{Assessment} = 1\)) or not (\(D_{Assessment} = 0\)).

\begin {align} D_{Assessment} = 1 \iff \theta _{Assessment} \geq T \nonumber \\ D_{Assessment} = 0 \iff \theta _{Assessment} < T \label {eq1} \end {align}

Researchers may then determine the diagnostic ability of this scale by comparing these classifications to those obtained from the gold standard (e.g., the misophonia questionnaire severity item; Wu, Lewin, Murphy, & Storch,  2014). Ideally, scores from the DMQ (\(\theta _{Assessment}\)) will correlate strongly with scores from the gold standard; however, prior research indicates that the strength of this correlation may vary (Bain, Norris, Conley, Manapat, & Ethridge,  2025). Moreover, there is often a direct relationship between classification performance and the strength of the correlation between \(\theta _{Assessment}\) and the gold standard (Gonzalez,  2021). As such, misclassification is inevitable (Figure 1). Researchers aim to maximize both sensitivity (true positive rate) and specificity (true negative rate), but sensitivity and specificity have an indirect relationship, such that as one increases, the other decreases (Gonzalez,  2021; Youngstrom,  2013). Decreasing \(T\) would increase sensitivity but decrease specificity. Increasing \(T\) would have the opposite effect (low sensitivity and high specificity). Decision theory techniques, which allow researchers to manually weight the importance of sensitivity and specificity, are recommended to determine the optimal cut-point (Smits, Smit, Cuijpers, & De Graaf,  2007). The following sections detail IRT and RF approaches to diagnostic assessment.

Item Response Theory

IRT is a collection of latent variable models that seek to uncover the underlying process that influences responses to observed variables (Edwards,  2009). They model the probability of endorsing a particular response on an item as a function of the latent trait (\(\theta \)) and item properties (Embretson & Reise,  2000; Lord,  2012). These properties include the difficulty of the item (i.e., threshold; \(b\)) and the item’s ability to discriminate between individuals (i.e., slope; \(a\)). Individual scores (\(\theta \)) are estimated using an individual’s responses and these item parameters (Edwards,  2009). IRT calibration is advantageous as it provides standardized, interpretable scores with greater variability than summed scores, conditional standard errors, and straightforward score equating (De Ayala,  2009; Embretson & Reise,  2000).

Samejima’s (1997) graded response model (GRM) is the most widely used IRT model for scales with items that consist of two or more ordinal response options (e.g., Likert scales). The GRM is expressed as: \begin {equation} P(x_j = c \mid \theta ) = \frac {1}{1 + \exp [-a_j(\theta - b_{cj})]} - \frac {1}{1 + \exp [a_j(\theta - b_{(c+1)j})]}, \label {eq2} \end {equation} which represents the probability of endorsing response option \(c\) given the latent trait (\(\theta \)), resulting in the observed response (\(x_j\)) to item \(j\). For more on the GRM and IRT in general, see work by Edwards (2009) or Embretson and Reise (2000).

The discrimination parameter (\(a\)) represents the slope of item \(j\) and indicates how strongly the item relates to the latent construct (i.e., a steeper slope reflects a stronger relationship). The threshold parameters (\(b\)) define the level of the latent trait required to endorse response \(c\) for item \(j\), with higher thresholds indicating higher trait levels (Manapat, Edwards, MacKinnon, Poldrack, & Marsch,  2021; Samejima,  1997). The GRM, like other parametric IRT models, assumes local independence, monotonicity, and unidimensionality. These assumptions enable interpretable item and person parameters but impose more structural constraints than nonparametric methods.

Differential Item Functioning’s Effects on Classification

While DIF is traditionally studied as a measurement bias problem, its presence can also affect classification accuracy. An item is said to exhibit DIF if two respondents from distinct subgroups (e.g., gender, primary language, etc.) have equal levels of \(\theta \) but different probabilities of endorsing a given response category for that item (Edwards,  2009). DIF indicates potential bias in items and threatens the validity of inferences drawn from test scores (Camilli & Shepard,  1994; Zumbo,  1999). There are two main types of DIF: uniform DIF and non-uniform DIF (Figure 2A and 2B, respectively). Uniform DIF occurs when the item uniformly requires a larger or smaller amount of the trait for one group, reflected by a difference in difficulty parameters but not in discrimination parameters (\(a_{Focal} = a_{Reference}\) and \(b_{Focal} \neq b_{Reference}\)). Non-uniform DIF occurs when the discrimination parameters differ between groups, meaning that the strength or direction of the group difference changes across the trait continuum (i.e., \(a_{Focal} \neq a_{Reference}\)). When classifying individuals using scores from scales with DIF, these group differences in item functioning can lead to systematic misclassification, particularly for members of the focal group (Gonzalez, Georgeson, Pelham, & Fouladi,  2021; Lai, Richardson, & Wa Mak,  2019; Paulsen et al.,  2020).

DIF can affect the overall scale score, resulting in differential test functioning (DTF; Chalmers, Counsell, & Flora,  2016; Yavuz Temel,  2023) when biased items disproportionately influence measurement for certain groups. For example, if items favor native English speakers, the scale score could lead to systematic misclassification for non-native speakers (Hambleton, Swaminathan, & Rogers,  2011; Zumbo,  1999). However, DIF does not always result in DTF (e.g., if only a few items have DIF or if the DIF items have low discrimination parameters). This underscores the importance of evaluating a scale for both DIF and DTF (Camilli & Shepard,  1994).

Random Forest

Random forests (RF) are an ensemble learning approach that improve prediction accuracy by aggregating the predictions of many decision trees built on bootstrapped samples of the data (Breiman,  2001). The RF additionally introduces randomness by selecting a subset of potential predictors at each split (Breiman,  2001; Jacobucci, Grimm, & Zhang,  2023). Specifically, for each of the decision trees, a bootstrap sample of size \(N\) is drawn from the data, which introduces sample-level randomness. A decision tree is then grown on each sample by recursively splitting the data using a randomly selected subset of \(m\) predictor variables at each node, which introduces variable-level randomness. This randomness helps reduce variance and improve generalizability to new samples. After all trees are trained, predictions are aggregated by averaging (for regression) or majority vote (for classification) across all trees.

The two primary tuning parameters for RF are the number of trees (\(B\)) and the number of predictors considered at each split (\(m\)). Typically, \(B\) is large, falling between 500 and 2,000. Increasing \(B\) improves model stability but increases computational cost. The variable subset size, \(m\), is recommended to be small (e.g., \(\sqrt {p}\), where \(p\) is the number of predictor variables). Decreasing \(m\) reduces the correlation between trees and can improve model performance by increasing the diversity of the ensemble (Breiman,  2001). Other RF tuning parameters include maximum tree depth, minimum node size, and impurity criterion (e.g., Gini or entropy). While these typically have less influence than the number of trees and predictors per split, they can affect model complexity and computational efficiency and may be adjusted depending on the application. For a full list of parameters, see Breiman (2001).

As the original RF algorithm assumes balanced data (i.e., equal proportions of classes), modifications need to be made when data are unbalanced. One such modification is the use of class-specific weighting. In this approach, larger weights are assigned to observations from minority classes during tree construction, which alters both the bootstrap sampling and the splitting criterion. By penalizing misclassification of minority-classes more heavily, weighting directly increases their influence on decision boundaries and reduces the tendency of RF to favor the majority class. This makes weighting particularly useful in diagnostic applications, where the minority class often represents the group of individuals in need of clinical intervention.

This study proposes using RF when DIF is present but unmodeled. RF has been shown to handle complex, high-dimensional data effectively, making RF suitable for psychometric applications where item responses may exhibit nonlinear relationships or interactions (Strobl, Hothorn, & Zeileis,  2009). Single-group IRT models assume item-invariance across subgroups (i.e., no DIF); however, the IRT framework can be extended to model DIF through multiple-group approaches when the cause of DIF is known. RF makes no such assumptions about item functioning. As a nonparametric method, RF can flexibly adapt to group-specific response patterns through its ensemble of trees. This flexibility may allow RF to maintain classification accuracy even when items function differently across subgroups, making it a potentially robust classification approach when DIF is unmodeled (e.g., DIF is suspected, but the source or structure of DIF is unknown).

1.2 Present Study

The current study aims to compare the performance of RF and IRT for classification in conditions when DIF is absent and present. We hypothesize that RF will demonstrate greater classification robustness in the presence of DIF because it can capture complex response patterns without assuming item-invariance. A secondary aim was to assess the impact of DIF on IRT-based classification performance. For example, we examine whether single-group IRT can maintain acceptable classification accuracy when DIF is minimal. We compare IRT, RF with predicted probabilities, and RF with predicted classifications in terms of accuracy, sensitivity, specificity, precision, negative predictive value (NPV), and F1 score (i.e., the harmonic mean of precision and sensitivity). We were also interested in the extent to which the classification performance of these models depended on features of the data. Importantly, all approaches (IRT-based, RF probability-based, and RF classes) did not have “access” to any DIF grouping variable, reflecting what applied researchers would likely do when DIF sources are unknown, unmeasured, or too complex to model using multi-group IRT approaches. In this respect, our simulation was designed to reflect common applied settings where researchers are likely to fit a misspecified single-group model. The simulated two-group DIF structure (discussed below) represents the simplest case of a broader set of scenarios that fail to account for DIF.

2 Methods

A Monte Carlo simulation study was conducted to examine the performance of IRT and RF in classifying (i.e., diagnosing) individuals when DIF in items is not directly identified and specified. Conditions with no DIF were included to serve as a baseline comparison. All data generation and analyses were conducted in R (R Core Team,  2024), and R scripts used for the simulation have been made publicly available on GitHub (https://github.com/cbain1/difml_open). We manipulated the following features of the generating model: sample size of the reference group, ratio of focal group to reference group, prevalence rate of the diagnosis according to the gold standard, number of items, DIF patterns, and magnitude of DIF.

2.1 Data Generation

The reference group size was set to 400 (small) or 2000 (large). The small sample size was meant to simulate datasets that are common in psychological research (Shen et al.,  2011), while the larger sample size was based on previous simulations examining DIF (e.g., Classe & Kern,  2024). The length of the scale was set to either 40 or 80 based on the lengths of published diagnostic scales (Raskin & Terry,  1988; Rosenthal et al.,  2021). The prevalence rate of the disorder was set to either 12 or 30, representing prevalence rates of disorders seen in the literature (Rosenthal et al.,  2021; Sims, Michaleff, Glasziou, & Thomas,  2021; Terry-McElrath & Patrick,  2018).

The ratio of focal group to reference group was set to either 1:1 (equal), 1:2, or 1:4; these values were chosen based on previous research (e.g., Classe & Kern,  2024). We chose a two-group DIF implementation that represents the simplest case for the research questions posed here. We recognize this is an “ideal” case and that the present study does not include conditions that reflect more complex DIF that is likely to be encountered in real data. However, the current simulation does reflect the approach applied researchers would likely take in situations where the grouping variable causing DIF is unmeasured or unknown.

In each simulation repetition, data were generated by a single continuous variable (\(\theta _{Diagnosis}\)) from a normal distribution with \(\mu =0\) and \(\sigma =1\) representing the latent variable determining the respondent’s true diagnosis (Equation 1; Gonzalez,  2021). To determine a threshold (\(T_{Diagnosis}\)) that would ensure our desired prevalence rate, we determined the \(1-p\) quantile of the \(\theta _{Diagnosis}\) distribution, where \(p\) represents the desired prevalence rate. Using this \(T_{Diagnosis}\) and Equation 1, the true diagnosis labels were generated. To examine the effects of impact (i.e., true group differences in \(\theta \)), a set of additional simulations were performed manipulating the data-generating distribution for the focal group. In these conditions, \(\theta _{Diagnosis}\) for the reference group simulees was generated from a normal distribution with a \(\mu =0\) and \(\sigma =1\), whereas \(\theta _{Diagnosis}\) for the focal group simulees was generated from a normal distribution where \(\mu =\{-1,1\}\) and \(\sigma = 1\).

The \(\theta _{Diagnosis}\) was then used with the GRM (Equation 1) to generate item responses. Based on previous research, \(a\)-parameters were drawn from a normal distribution with \(\mu = 1.7\) and \(\sigma = 0.3\) (Hill,  2004; Manapat & Edwards,  2022). Also based on previous research, \(b\)-parameters were generated by drawing \(b_1\) from a normal distribution with \(\mu =-1.5\) and \(\sigma = -0.5\) (Hill,  2004; Manapat & Edwards,  2022). Then, a difference parameter was drawn from a normal distribution with \(\mu =1\) and \(\sigma =0.2\), and added to \(b_1\) to create \(b_2\) (see Manapat & Edwards,  2022). This process was repeated until a total of four \(b\)-parameters were created (e.g., \(b_2\) = \(b_1\) \(+\) difference parameter 1).

To simulate DIF, we manipulated DIF proportion, type of DIF, strength of DIF, and whether DIF was balanced by changing the focal group parameters. DIF was set to be present in 0, 5, 40, and 70 percent of the items based on previous published DIF research (Classe & Kern,  2024; Patel, Robison, & Cougle,  2024; Paulsen et al.,  2020). In conditions with DIF in at least five percent of items, DIF was simulated in either the \(a\)-parameter, the \(b\)-parameter, or both parameters. DIF in the \(a\)-parameter was simulated by multiplying the \(a\)-parameter of the reference group by a constant, \(h\) = 0.2, 0.5, or 0.8. DIF in the \(b\)-parameter was simulated by changing the \(b\)-parameter of the reference group by a constant \(s\) = 0.5, 1.0, or 2.0. In cases where DIF was unbalanced, all parameters were increased multiplicatively for the \(a\)-parameter or additively for the \(b\)-parameter. In conditions where DIF was balanced, half the items with DIF were increased, while half were decreased. To determine these values for \(h\) and \(s\), a small simulation was performed to ensure that some levels of \(h\) and \(s\) would result in visible changes in the TIFs (i.e., DTF) while others would not. The results from this simulation can be seen in Supplementary Figure 1.

Example category characteristic curves from the five simulation conditions with the most extreme differences between the focal and reference groups can be found in Figure 3. These examples illustrate the true parameter differences between groups (dotted and dashed lines) as well as the biased parameters estimated using the misspecified single-group GRM (solid lines). The solid lines demonstrate how model misspecification leads to biased parameter estimates that inadequately represent either group’s true response patterns and, in turn, biased \(\theta \) estimates of an individual’s level of the latent trait. They also highlight the practical consequences of failing to account for group differences in item functioning. Example distributions of both the generating \(\theta \) and estimated \(\theta \) from the same five simulation conditions can be found in Figure 4. Estimated \(\theta \) values were obtained via expected a posteriori (EAP) scoring from the misspecified single-group GRM fit to pooled data. These illustrate the bias in \(\theta \) estimates resulting from the misspecified single-group GRM fit to pooled data when DIF is ignored (dashed lines). Together, Figures 3 and 4 illustrate both the item-level source and the person-level consequence of model misspecification under unmodeled DIF.

All non-DIF-related manipulated variables were fully crossed to create a total of 24 simulated conditions with no DIF. A total of 432 conditions with only \(a\)-parameter DIF were generated. An additional 432 conditions with only the \(b\)-parameter DIF were generated. When DIF was present in both the \(a\)- and \(b\)-parameters, all conditions were fully crossed to generate 1,296 conditions. Item responses were generated in R (R Core Team,  2024), with 1,000 data sets generated for each condition. All program specifications were set to the defaults. Of the 2,184 conditions, 24 did not contain DIF, 456 contained DIF with DTF, and 1,704 contained DIF without DTF. The additional conditions with impact were fully crossed with the 456 DIF conditions resulting in DTF. This resulted in an additional 912 conditions. As such, a total of 3,096 conditions were examined.

2.2 Classification Models

The simulated datasets were analyzed with one psychometric approach, IRT, and one nonparametric approach, RF. For the IRT model, the estimated \(\theta _{Diagnosis}\) were obtained using IRT expected a posteriori (EAP) scores. Thresholds for IRT were determined through ROC curve analysis (Brown et al.,  2001; Florkowski,  2008), discussed in the following section. RF does not require a threshold.

Determining Cut-Points and Probability Thresholds

In ML, a default probability cut-point of .50 is often set, such that cases with a predicted probability of diagnosis of 0.5 or higher would be classified as diagnosed (Gonzalez,  2021). This approach is referred to as using a Bayes classifier. However, this approach does not allow researchers to consider the relative importance of sensitivity and specificity within their research contexts. For example, when evaluating a screening tool for a disorder, researchers may wish to prioritize sensitivity over specificity, as they would rather send individuals on for (potentially unnecessary) further testing than miss individuals who would benefit from treatment (e.g., see Graham et al.,  2019, for a discussion of this approach with an eating disorder measure) . However, other psychological studies have prioritized specificity over sensitivity (Parker-Guilbert, Leifker, Sippel, & Marshall,  2014; Wardell et al.,  2020). See the following for discussions of the ethical issues of false positives in clinical psychology (Bradford, Meyer, Khan, Giardina, & Singh,  2024; Wakefield,  2010, 2015).

Determining a threshold with an ROC curve allows a researcher to prioritize sensitivity, specificity, or balance their importance. ROC curves illustrate the changes in sensitivity and specificity that occur as a function of the threshold. Sensitivity and specificity are calculated for each cut-point, and these values are then plotted, forming the ROC curve. For example, if researchers wanted a classifier that balances sensitivity and specificity, researchers could choose the threshold that maximizes the Youden index (X. Liu,  2012). The Youden index corresponds to the point on the ROC curve that is farthest from the chance diagonal. In practice, researchers may not value sensitivity and specificity equally and may wish to choose the threshold that prioritizes one over the other. Once the cut-point has been set, the cut-point is then imposed on the \(\theta \) estimates to get the predicted diagnosis. Given that we have no a priori reason to prioritize one over the other in this study, we have chosen the thresholds for the IRT models based on maximization of the Youden index, as was done in previous research (Gonzalez,  2021).³

One can also assess how well the diagnostic scale discriminates via the shape and height of the ROC curve. The area under the ROC curve (AUC) can also help compare the performance of classifiers. The AUC measures the likelihood of making a correct classification when one observation from each group (e.g., one individual who is diagnosed and one who is not) is chosen at random. It can also be interpreted as the average sensitivity across all values of specificity, or the reverse, the average specificity across all values of sensitivity. A curve that follows the chance diagonal will have an AUC of .50, while curves that illustrate stronger performance will have AUCs closer to 1.0.

Role of Training and Test Data

Split-half cross-validation was implemented so that 50% of the sample was used to train the model (training set), and the remaining 50% was used for evaluation (test set). Sparseness checks were performed on the training data to ensure that each item response category was endorsed at least five times across all items to avoid zero or near-zero cases, as was done in previous IRT simulations (Gonzalez,  2021; Manapat & Edwards,  2022). If this was found not to be the case, this data set was discarded and resampled until the sparseness conditions were met. For the IRT models, the training set was used for item calibration and to determine the cut-point using ROC curves. The RF models were trained on the training set and evaluated on the test set to predict diagnostic classes. For the RF models, all tuning parameters were set to defaults, consistent with recommendations from previous research (Breiman,  2001; Gonzalez,  2021).

2.3 Assessing Classification

We evaluated the performance of the two classification approaches using a combination of metrics that assess different dimensions of classification quality: (a) accuracy which measures the proportion of correctly classified instances (both true positives and true negatives) out of the total instances, (b) sensitivity which quantifies the models’ ability to correctly identify positive instances, (c) specificity which measures the models’ ability to correctly identify negative instances, (d) precision which assesses the proportion of true positive predictions among all positive predictions, (e) NPV which quantifies the proportion of true negative predictions among all negative predictions, and (f) F1 score, the harmonic mean of precision and sensitivity, providing a balanced measure of the models’ performance. The goal was to assess how well the models could correctly classify simulees into their generating categories.

Effect Sizes for Classification Metrics

Results were first examined visually. Then, ANOVAs were conducted to assess the impact of manipulated data features (sample size of the reference group, ratio of focal to reference group, prevalence rate of the diagnosis according to the gold standard, number of items, DIF patterns, and magnitude of DIF) and the classification techniques (IRT and RF) on all classification outcomes. Main effects and interactions up to three-way interactions were included in the analysis. To ensure that the most influential effects were of focus, we probed all effect sizes (partial \(\eta ^2s\)) greater than or equal to .06 (medium effect; Cohen,  2009). All other effects were omitted.

3 Results

Given the 3,096 unique data conditions, we present a subset of results that focuses on key trends and significant interactions. The selection of conditions was guided by the magnitude and consistency of observed effects, with an emphasis on medium or large effect sizes (\(\eta ^2 \geq .06\); Cohen,  2009) and interactions that exhibited divergent method performance. The results are organized according to the presence of both DIF and DTF. We first discuss the 24 conditions where DIF was not present in the data, followed by conditions with DIF with DTF, those with DIF without DTF, and finally, the additional conditions where impact was introduced.

3.1 No DIF Conditions

ANOVA models were used to analyze simulation results. The type of approach was influential on all classification measures (Table 1). IRT was found to produce models with higher specificity and precision, while RF produced models with higher accuracy, sensitivity, NPV, and F1 scores. It is worth noting, however, that many of these differences were quite small, occurring in the third or fourth decimal place (Table 1). Thus, there were no practical differences in performance across conditions. These results align with previous findings, supporting that IRT and RF are equally good at classifying simulees based on diagnostic assessments without DIF (Gonzalez,  2021). However, these results expand upon previous findings, illustrating the strengths of each approach on different types of classification measures (e.g., sensitivity versus specificity).

3.2 DIF Conditions without DTF

Conditions that contained DIF without DTF were analyzed, but no meaningful differences were found between the pattern of results for these DIF conditions and no DIF conditions. However, the inclusion of DIF did result in more unstable performance across replications. As such, results of conditions with DIF but no DTF are not included here but can be found in the Supplementary Materials.

3.3 DIF Conditions with DTF

As we were interested in examining potential differences in classification performance across methods (IRT versus RF) in simulees in the focal group versus the reference group, ANOVAs were conducted as follows for each outcome. First, we fit a model using group (focal versus reference) and all manipulated data features (total sample size, ratio of focal to reference group, number of items, prevalence rate, proportion of items containing DIF, whether DIF was balanced, and strength of both threshold and slope DIF) as main effects. Variables with partial \(\eta ^2 \geq .06\) (medium effect; Cohen,  2009) were deemed influential. Then, to create final models, two- and three-way interactions were probed that contained at least one influential variable. Interaction effects were deemed influential if they had partial \(\eta ^2 \geq .06\) (medium effect; Cohen,  2009). The classification method was found to be influential for all outcomes. Effect sizes of all influential effects and the mean performance of each classification method can be found in Table 2.

Effects of Data Characteristics

We first examined the influential effects of data characteristics, as presented in Figure 5. We found that higher prevalence rates, or more balanced class sizes, led to lower precision in the focal group (5A) but no change in the reference group. The specificity of focal group classifications was also found to decrease as more items contained DIF (5B), while specificity of reference group classifications slightly increased. Sensitivity of reference group classifications decreased as the proportion of focal group members increased (5C). The same pattern was observed for NPV. This decline likely reflects a shift in the classification model’s estimates toward the focal group, as model training was based on the combined sample.

Precision remained stable across varying levels of threshold DIF strength when the focal group contained 50% of the total sample. However, when the focal group represented a smaller portion of the sample, precision declined as the strength of threshold DIF increased (5D). Across all sample compositions, the F1 score decreased as threshold DIF strength increased, with the steepest decline occurring when the focal group comprised half of the overall sample (5E). Lastly, the overall accuracy of all models decreased as more items contained DIF (5F).

Effects of Classification Method.

Next, we shift our focus to the effects resulting from method type (IRT vs. RF) on each classification metric.

Accuracy. The proportion of the sample that consisted of the focal group, or the proportion of simulees in the sample with DIF, did not influence the accuracy of RF models but did influence the accuracy of IRT models such that a larger focal group led to lower accuracy (Figure 6A). Increases in threshold DIF strength, or how much more severe symptoms of simulees in the focal group must be compared to the reference group, led to lower accuracy in IRT (6B), whereas RF models were relatively robust to changes in threshold DIF.

Sensitivity. Across all threshold DIF strengths and all examined focal group percentages, RF models had higher sensitivity than IRT models (Figure 7A). The sensitivity of IRT models decreased as the strength of threshold DIF increased, with the strongest effect occurring when at least 50% of the sample was in the focal group. Increases in threshold parameter DIF did not affect the sensitivity of RF models but led to decreases in sensitivity for the reference group of IRT models (7B). No visible differences were observed in the sensitivity of focal group classifications across RF and IRT models. However, since reference group sensitivity was worse in IRT models than RF models, the overall sensitivity of RF models was greater than that of IRT models.

Specificity. Influential method effects on specificity are presented in Figure 8. The specificity of IRT models was influenced by both the strength of threshold DIF and the size of the focal group (8A). When the focal group constituted 50% of the overall sample, IRT models were robust to varying threshold DIF strength and exhibited superior specificity compared to RF models. However, when the focal group comprised 20 or 30 percent of the sample, IRT model specificity declined as threshold DIF strength increased. In cases where threshold parameter DIF exceeded one and a half, RF models outperformed IRT models in terms of specificity.

This effect is further illustrated in Panel 8B, where we see that the specificity of the reference group remained relatively stable across varying threshold DIF strengths for both IRT and RF models, with minor increases observed as the strength increases. Conversely, focal group specificity was substantially affected by threshold DIF strength, and this relationship diverged between the two modeling approaches. For IRT models, specificity consistently declined with increases in threshold parameter DIF. RF models also exhibited decreasing specificity up to a threshold DIF strength of one, after which the specificity stabilized. Consequently, under high \(b\)-parameter only DIF conditions, RF models yielded higher specificity for focal group members and the overall sample than IRT models. In contrast, when threshold DIF strength was low, IRT models maintained higher specificity.

Panel 8C illustrates the varying effect the subsample and focal group proportion had on specificity across model types. Specifically, reference group specificity increased as the focal group proportion increased in both IRT and RF models. This likely resulted from consequential sample size increases, with IRT models always producing higher reference group specificity than focal group specificity. When the sample contained equal numbers of reference and focal group simulees, IRT models resulted in higher focal group specificity than RF models, but if the focal group was less than half the total sample, RF models had higher focal group specificity than IRT models. If examining the overall sample only, RF and IRT models yielded equal specificity values, on average, until the focal group comprised at least 40% of the overall sample, when IRT began to yield better sensitivity than RF models.

Precision. The reference group precision of both IRT and RF models was robust to changes in the strength of threshold DIF, with IRT slightly outperforming RF models. Both RF and IRT models had worse focal group precision as the strength of threshold DIF increased, with strength of threshold DIF having a stronger effect on IRT models than RF models, such that as threshold parameter DIF strength reached or exceeded one, RF outperformed IRT in the focal group. This result held in the full sample results as well.

NPV. Regardless of sample type, RF models had higher NPV values than IRT models, and the NPV of RF models was robust to changes in focal group proportion (Figure 9A). However, for IRT models, the NPV of the reference group decreased as more of the sample contained simulees experiencing DIF, which in turn decreased the NPV of the full sample in IRT models. We saw a similar influence of threshold DIF strength (9B), where RF models were robust to changes in threshold DIF strength, regardless of the sub-sample, while IRT models had lower NPV values as threshold parameter DIF strength increased.

F1 Score. RF models were found to produce higher F1 scores than IRT models, regardless of either threshold DIF strength or the proportion of the the sample that consisted of the focal group. In addition, threshold DIF strength and proportion of the the sample that consisted of the focal group did not affect the F1 score of the RF models, whereas increases in either led to worse F1 scores in IRT models.

3.4 Impact Conditions

As we were interested in examining potential differences in classification performance across methods (IRT-based versus RF-based) in simulees in the focal group versus the reference group, ANOVAs were conducted as follows for each outcome. First, we fit a model using group (full, focal, and reference) and all manipulated data features (total sample size, ratio of focal to reference group, number of items, prevalence rate, proportion of items containing DIF, whether DIF was balanced, strength of both threshold and slope DIF, and impact) as main effects. Variables with partial \(\eta ^2 \geq .06\) (medium effect; Cohen,  2009) were deemed influential. Then, to create final models, two- and three-way interactions were probed that contained at least one influential variable. Interaction effects were deemed influential if they had partial \(\eta ^2 \geq .06\) (medium effect; Cohen,  2009). Effect sizes of all influential effects and the mean performance of each classification method can be found in Table 3. The classification method was found to be influential for all outcomes except for specificity and precision. Note that this is the same as results in non-impact conditions (Table 2).

As many effects were the same between Table 2 and Table 3, we focus our discussion here on the three unique effects in Table 3. We begin with the two effects that did not contain impact. First, it was observed that as more items contained DIF, the specificity of the model decreased (Figure 10A). Second, as the strength of DIF in the \(b\)-parameter increased, the specificity of both the full sample and focal group decreased, with this effect being stronger in the focal group (10B). However, the specificity of the reference group improved as the strength of DIF in the \(b\)-parameter changed from zero to one and then began to decrease slightly once DIF strength surpassed one. This disordinal interaction is interesting and could be the result of the relationship seen between method and \(b\)-parameter DIF in Figure 7A. Lastly, when impact was either zero (focal and reference group ability come from the same underlying distribution) or positive (focal group \(\mu = 1\), reference group \(\mu = 0\)), the specificity of the models appears to be relatively unaffected (10C). However, when the impact is negative (focal group \(\mu = -1\), reference group \(\mu = 0\)), the specificity of the focal group across methods is notably lower (mean = 0.51) than that of the full sample (mean = 0.90) and reference group (mean = 0.95). Note that no influential interaction effect on specificity was found between sample type, impact, and method (partial \(\eta ^2\) = 0.01), indicating that both RF and IRT have poor focal group specificity when impact is negative (i.e., the mean ability of the focal group is notably lower than that of the reference group).

4 Empirical Example

To illustrate the practical application of IRT and RF classification methods in the presence of DIF, we applied both approaches to data collected via the University of Oklahoma’s Psychology Research Pool. The purpose of the study was to predict misophonia diagnosis based on item responses to the DMQ (Rosenthal et al.,  2021). As misophonia is often co-diagnosed with autism, we hypothesized that DIF may occur across gender in misophonia, as is the case in autism (Schiltz & Magnus,  2020). Informed consent was obtained electronically, and all procedures were approved by the Institutional Review Board. After removing participants who failed attention checks, were under 18 years old, or reported a gender other than cisgender man or cisgender woman, the final sample included 725 participants (80.3% female, 75.6% White, \(M_{age}\) = 18.8, \(SD_{age}\) = 1.28). Male participants were less likely to pass attention checks than female participants (\(\beta \) = -0.54, \(SE\) = 0.22, \(z\) = -2.49, \(p <\) .05). No missing data were present in the sample. For more on this sample, see Bain et al. (2025).

4.1 Measures

DMQ

The 12-item impairment subscale of the DMQ (Rosenthal et al.,  2021), which assesses the extent to which auditory triggers interfere with daily functioning, was used in this study. Items use a 5-point Likert-type scale ranging from 0 (Not at all) to 4 (Extremely). Gender-based DIF was identified in four of these 12 items (see Supplementary Figure 2 for trace lines).

Misophonia Questionnaire (MQ)

The MQ severity question (Wu et al.,  2014) asks participants to rate their sound sensitivity from 1 (minimal) to 15 (very severe). As is traditionally done, this item was used as a self-report gold standard to split participants into diagnostic groups based on a threshold cutoff value of seven or above.

Classification Models

Both IRT and RF models were evaluated using the DMQ impairment items as predictors and the MQ severity-based diagnosis as the outcome. For the IRT model, \(\theta \) estimates were obtained using the GRM, and diagnostic classifications were made based on ROC-optimized thresholds using the Youden Index. For the RF model, classification was performed using a Bayes classifier. All models were evaluated in a 50% hold-out (i.e., test) dataset using the classification metrics described previously.

4.2 Results

Classification Outcomes

Classification performance for IRT and RF models is shown in Figure 11 across the total sample and by gender. Overall, RF outperformed IRT on key metrics including accuracy, F1 score, sensitivity, and NPV, suggesting RF more effectively identified both diagnosed and non-diagnosed simulees. These advantages held across the total sample as well as the female and male subgroups. In contrast, IRT yielded higher specificity and precision, indicating a more conservative approach with fewer false positives but also reduced sensitivity, particularly among females. This trade-off mirrors patterns observed in the simulation study, where IRT underperformed in the presence of DIF. These results highlight the strength of RF in diagnostic contexts where sensitivity is prioritized or DIF may be unaccounted for.

Variable Importance

Although RF does not provide the same, clearly interpretable relationship between the classifications and item responses as IRT, RF does output variable importance scores. One such variable importance score is the mean decrease in accuracy, which reflects how much the model’s overall prediction accuracy decreases when that variable is systematically altered (i.e., randomly permuted) to remove its association with the outcome variable. We examined the relative importance of DIF items versus non-DIF items in the RF model and found that the two most important variables (which together contribute 47.38% of the total variable importance) in the RF model were DIF items. All item importance scores can be found in Supplemental Table 4.

5 Discussion

The current study sought to evaluate how DIF impacts classification accuracy using parametric (IRT) and nonparametric (RF) models. Accurate classification in psychology is paramount, yet classification accuracy may decrease when DIF is present but unmodeled, leading to erroneous group comparisons, compromised construct validity, and issues with replication. Given the increasing use of machine learning in psychological science, this study investigated whether RF maintains more robust classification performance than single-group IRT when DIF is present but not explicitly modeled. The empirical results from the misophonia dataset mirrored these simulation-based trends, indicating that, the advantages of RF observed in the simulations generalize to real-world diagnostic contexts. Notably, RF achieved greater classification stability across gender groups, whereas IRT demonstrated lower sensitivity among females, suggesting that unmodeled DIF can meaningfully reduce classification accuracy in practice.

5.1 Parametric Assumptions and Classification Under Non-DIF Conditions

When DIF was absent, both IRT and RF showed comparable average performance across classification metrics. These findings are consistent with prior research showing that parametric and nonparametric classifiers perform similarly when assumptions are met (Gonzalez,  2021). However, IRT models make a number of critical assumptions (e.g., monotonicity, local independence, item invariance, and unidimensionality). Violations of these assumptions, particularly in applied data where multidimensionality or local dependence is likely, can reduce the classification performance of IRT-based approaches. RF, which does not rely on these assumptions, showed marginally higher accuracy, sensitivity, NPV, and F1 scores, especially as sample size increased. These differences, while marginal, point to a broader tradeoff. IRT provides a theory-driven framework for modeling latent traits and item functioning, but its parametric assumptions must be reasonably met for accurate inference. RF offers a flexible, data-driven alternative with advantages when data characteristics substantially violate IRT assumptions. However, RF does come with the cost of reduced interpretability, which is potentially problematic when explanation and transparency are priorities. It is important to reiterate that our study compares RF to single-group IRT. More complex IRT-based approaches, including multiple-group IRT (when the cause of DIF is known) and mixture IRT (when the cause of DIF is unknown and impact is present), can explicitly model DIF but cannot always reasonably be implemented. Our findings therefore reflect the performance trade-offs between RF and single-group IRT in scenarios where DIF exists but is ignored or cannot be modeled.

5.2 The Influence of DIF: Classification Stability and Robustness

When DIF was present but not strong enough to produce DTF, both IRT and RF maintained performance close to what was seen in conditions with no DIF. However, greater variability was observed across simulation replications, particularly for IRT. This suggests that unmodeled DIF, even when not resulting in DTF, can reduce the stability of classification decisions.

Under more severe DIF conditions involving DTF, IRT’s classification performance decreased (as expected). Decreases in performance were systematic and not limited to the focal group, reflecting broader model misspecification. RF, in contrast, maintained relatively stable performance despite increases in DIF item count, magnitude, or focal group size. This robustness suggests that RF’s flexible modeling of item interactions allows it to maintain classification performance when DIF is present but not modeled. These simulation findings were further supported in the empirical example, where gender-based DIF in the DMQ items reduced IRT sensitivity but had a negligible impact on RF classification. Despite clear item-level DIF, RF maintained high diagnostic accuracy and F1 scores across subgroups, consistent with the simulated evidence that RF is robust to latent sources of DIF and DTF. This consistency across simulated and empirical data further demonstrates RF’s practical robustness to DIF effects.

RF’s classification robustness under DIF conditions may stem from its ability to adaptively partition the response space. The fairness literature in ML suggests that ensemble methods like RF can learn proxy variables for latent group membership, thereby partially mitigating the impact of unmeasured DIF (Classe & Kern,  2024; Kraus, Wild, & Hilbert,  2024). RF can maintain classification performance by partitioning on combinations of item responses that reflect underlying group differences without requiring explicit identifications of those groups. The present data offer some evidence for this. In the empirical example, the two items with the highest variable importance scores were both DIF items, which is consistent with the idea that RF prioritized items containing group-specific signal to maintain classification performance. However, this finding comes from a single dataset and does not constitute a formal assessment of what aspects of the RF algorithm allow it to maintain classification performance. Whether RF is explicitly partitioning on item combinations that proxy DIF group membership remains an open question.

It is worth noting that this finding may raise concerns when the training data contains biased or noisy labels (e.g., Garb,  2021). If the gold standard diagnosis reflects systematic misclassification (e.g., underdiagnosis in marginalized populations), RF may learn and reproduce these patterns in its classifications. This introduces the risk of reifying diagnostic bias, particularly if the model is used without adequate scrutiny or interpretation tools. As such, we recommend that researchers carefully evaluate the quality and potential bias of gold standard classifications and utilize post hoc interpretation methods in contexts where model decisions have direct implications for diagnosis, treatment, or resource allocation.

5.3 The Role of Impact on Classification Performance

The introduction of impact (i.e., group-level mean differences in the latent trait) had influences on classification performance that extend beyond those observed under non-impact conditions alone. Notably, the presence of negative impact (i.e., when the focal group’s mean trait level is lower than that of the reference group) led to a pronounced decrease of focal group specificity, regardless of whether IRT or RF was used. This finding echoes concerns in the fairness literature, where group-based disparities in base rates or latent trait distributions can produce disproportionate false positive or false negative rates, even in the absence of overt bias in the model itself (Garb,  2021). From a practical standpoint, this indicates that more focal group members were incorrectly classified as having the condition, which could result in overdiagnosis in real-world settings. Importantly, this pattern did not interact significantly with the classification method, suggesting that both IRT and RF are affected by negative impact. This finding represents a limitation to RF’s robustness: while it maintains classification performance in the presence of unmodeled DIF, it cannot fully compensate for true group-level differences in trait distributions. In other words, RF can maintain classification performance despite DIF, but disparities that result from true trait differences rather than item-level bias will still be present in RF classifications. This underscores the need to distinguish between bias (e.g., DIF) and true differences (i.e., impact) when evaluating classification fairness and utility.

5.4 Interpretability, Fairness, and Clinical Implications

While RF demonstrated stronger classification performance under DIF and DTF conditions, its limited interpretability should prompt careful consideration when used in clinical settings. In clinical contexts, where classification outcomes may inform diagnosis or treatment, stakeholders often need to understand how a decision was reached, not just whether it was accurate. RF’s flexibility enables it to model complex response patterns and adapt to unobserved sources of bias, but the internal logic of ensemble methods is relatively unclear. As a result, it may be difficult to determine whether maintained classification performance reflects appropriate adaptation to complex response patterns or a reliance on spurious associations in the data.

As RF is an ensemble method, the decision rules for creating model predictions are unknown, and RFs are not directly interpretable. Metrics like variable importance measures can be used to identify which variables (i.e., items) the model is most reliant on when creating predictions. In the empirical study, the two most influential items for diagnostic prediction were DIF items. This may indicate that these items, despite exhibiting DIF, capture aspects of misophonia most strongly predictive of diagnosis. However, this was only one circumstance for identifying how RF uses DIF versus non-DIF items in diagnostic classification. Future simulation research is needed to investigate the use of variable importance measures to determine the relative influence of DIF versus non-DIF items on diagnostic classification in RF. In applied settings, researchers should be sure to investigate whether the most important variables align with theoretical expectations or clinical relevance.

The inclusion of impact in our simulation design also reinforces the need for interpretability when group-level differences exist. Misclassification due to true group differences could have real-world consequences, particularly in clinical assessment, where over- or under-diagnosis may lead to inappropriate treatment or missed intervention opportunities. Post hoc interpretability tools and fairness evaluations are critical to understanding whether classification errors are distributed equitably across groups. In the context of parametric modeling, approaches like mixture IRT may offer a promising solution by explicitly modeling both DIF and impact within a unified framework, allowing researchers to understand whether classification errors stem from measurement bias versus true trait differences (Sajobi et al.,  2022). Simulation-based calibration or reweighting strategies could also be explored in future work to mitigate the effects of impact when ground-truth differences are known or suspected (Naderalvojoud, Curtin, Asch, Humphreys, & Hernandez-Boussard,  2025).

When grouping variables are known and measured, the appropriate approach is to test for DIF and use approaches like multi-group IRT or mixture IRT for scoring and classification. Our findings are not intended to replace these approaches. Rather, they address scenarios where traditional DIF modeling cannot be implemented—when relevant grouping variables are unknown, unmeasured, or too complex to model. In such cases, our results suggest RF may maintain more robust classification performance than single-group IRT and therefore may be a reasonable alternative approach to classification.

While RF may offer practical advantages in research contexts where maintaining classification performance under model misspecification is prioritized over interpretability (e.g., early-stage screening, exploratory identification of risk or symptom groups, large-scale accuracy-focused classification), its use in high-stakes clinical decision-making warrants caution. For example, when decisions must be justified, when treatment planning requires transparency, or when fairness and accountability are critical. Conclusions about clinical utility should acknowledge that simulation studies, while useful for testing specific conditions, cannot fully reflect the complexity of real-world diagnosis, including comorbidity, diagnostic ambiguity, and contextual influences.

5.5 Limitations and Future Directions

As with any study, there are limitations to consider. First, our simulations focused exclusively on unidimensional data structures. Although this serves as a necessary first step in isolating the effects of DIF on classification accuracy, many constructs in diagnostic assessment are multidimensional. Future work should examine the impact of DIF in multidimensional IRT and ML contexts to assess generalizability. Second, only a single ML approach (e.g., RF) was implemented in this study. While RF is widely used for its accuracy (especially in subsequent or new samples), examining only one nonparametric method restricts the scope of this simulation and may overlook important comparisons with other algorithms. We hypothesize that RF’s ability to model interactions between items may be critical. DIF creates complex patterns where the relationship between items and diagnosis depends on response patterns across multiple items. RF’s recursive partitioning may capture these patterns (i.e., interactions between items) without researchers having to explicitly specify interaction terms, as would need to be done in models like regularized logistic regression. Future research should systematically compare RF to other machine learning methods (e.g., regularized logistic regression, elastic net) to determine which specific algorithmic features maintain classification robustness under unmodeled DIF. For example, future research could examine the performance of decision trees that leverage the same strengths regarding item interactions but are more interpretable than RF models.

Third, the simulation used a two-group DIF structure, which does not fully capture the applied scenarios that motivated this study, where the source of DIF may be complex. The two-group design was chosen because it represents the simplest case, and the observed performance of RF relative to single-group IRT-based classifications may be a lower bound: as DIF structure becomes more complex, the bias in \(\theta \) estimates from the misspecified single-group IRT approach would almost certainly be worse. Future work should examine how these classification approaches perform when the complex scenario is directly simulated, that is to say, DIF should occur across multi-group, intersectional, or other complex groups, to determine whether the present findings generalize to these increasingly complex scenarios.

Fourth, although RF models can be partially interpreted through tools like variable importance scores or partial dependence plots, these do not guarantee actionable insight. For example, an item exhibiting DIF may appear less important in the RF model, but whether this reflects the model appropriately minimizing its influence or a confounding interaction remains unclear. Future research could evaluate variable importance within simulation frameworks to better understand how RF maintains classification accuracy when DIF is present. Lastly, future work should explore interpretable ML strategies, such as shallow decision trees, to help understand which response patterns RF uses to maintain classification under DIF conditions. For instance, one could train a decision tree with a maximum depth of two to capture simple interaction patterns in the data. The resulting terminal node memberships could then be used as indicators of latent grouping structures or proxy variables influencing DIF. These derived groups could subsequently be incorporated into a traditional IRT framework to model DIF more explicitly, offering a novel way to combine RF’s demonstrated classification performance with IRT’s interpretability.

Our comparison focused on single-group IRT and on polytomous data analyzed with the GRM. While the GRM is the most widely used IRT model for Likert-type scales in psychology, other IRT models (e.g., 1PL, 2PL, 3PL) are commonly used in educational assessment and for some psychological measures with binary response formats. Given that the GRM reduces to the 2PL when the number of response options is two and the 1PL is often considered a more constrained version of the 2PL, it is likely that the GRM represents a more complex and potentially challenging scenario for model fitting. Examining the GRM specifically serves as a reasonable starting point for this line of inquiry and provides a proof-of-concept for the approach. However, additional work is needed to generalize the results of the current study to dichotomous models, as the nature of DIF and the performance characteristics of both IRT and RF may differ when working with alternative IRT models. It would also be useful for future simulation studies to examine whether our pattern of results holds for other response formats that play a significant role in psychological assessment (e.g., forced-choice questionnaires such as the MMPI).

When DIF is suspected but groups are unknown, mixture IRT models and latent class analysis offer psychometric alternatives to model latent group structure. These approaches would be particularly relevant for comparison in impact conditions where groups differ in trait distributions, as class separation may be more identifiable under such conditions. Future research should compare RF to these mixture modeling approaches and examine whether our findings extend to other IRT models to provide a more complete picture of available methods for handling unmodeled DIF. This would help clarify whether RF’s robustness reflects a limitation of single-group IRT specifically or represents benefits relative to all available psychometric approaches. It is also worth noting that comparisons with mixture IRT would be particularly informative in the complex multi-group DIF scenarios not examined here. The two-group findings reported in the present study should be treated as initial evidence for rather than a complete demonstration of RF’s relative performance under unmodeled DIF.

Additionally, more research is needed to evaluate the generalizability of these findings for more data contexts that are representative of a larger number of psychological scales. For example, many real-world screeners are brief (e.g., 10–20 items). Although our empirical example uses a short scale (i.e., 12 items), future simulation work should investigate the performance of these methods when using shorter instruments. Future work should consider incorporating domain-specific interpretation methods to ensure that ML models align with the substantive research goals in psychology. For example, a domain-specific interpretation method could involve aligning the RF model’s item importance scores with clinically meaningful symptom dimensions, as determined by clinicians or existing diagnostic frameworks. Items identified as highly important by the model could be examined to determine whether they correspond to core features of the psychological construct being assessed (e.g., anhedonia in depression). This alignment would help researchers to interpret the model’s classification decisions in a manner consistent with diagnostic theory and practice, enhancing the clinical validity and utility of the ML outputs.

6 Conclusion

These findings contribute to the growing body of literature on nonparametric ML methods in psychometric contexts (Classe & Kern,  2024; Gonzalez,  2021; Kraus et al.,  2024; Orru, Gemignani, Ciacchini, Bazzichi, & Conversano,  2020; Quan & Wang,  2026). Our results demonstrate that RF models maintain stable classification performance under adverse data conditions, even when DIF is ignored. These findings suggest that RF models may serve as a valuable complement to parametric frameworks, particularly in applied contexts where fairness, generalizability, and classification reliability are critical. While RF offers empirical flexibility, it is not a substitute for IRT or more traditional psychometric models, which remain essential for applications requiring latent trait estimation, theoretical alignment, and construct interpretation. Rather than advocating for the replacement of theory-driven models, this study supports a context-sensitive and pluralistic approach to classification, drawing on the strengths of both parametric and nonparametric methods. As psychological assessment increasingly confronts complex and heterogeneous populations, aligning methodological choices with both theoretical and empirical demands will be crucial to ensuring validity, fairness, and replicability in diagnostic classification.

Notes

Correspondence concerning this article should be addressed to Catherine M. Bain and Patrick D. Manapat, Department of Psychology, University of Oklahoma, 455 W. Lindsey Street, Dale Hall Tower, Room 705, Norman, OK 73019, United States. Email: cbain1@ou.edu; pmanapat@ou.edu

Article Information

Conflict of interest disclosures

No potential conflicts of interest were reported by the authors.

Funding

The authors reported there is no funding associated with the work featured in this article.

Ethical principles

The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Data availability statement

The simulation R code relevant to this manuscript is published on GitHub: https://github.com/cbain1/difml_open. Empirical data are available upon reasonable request to the corresponding author.

Supplementary Materials

The supplementary materials for this article are published on GitHub: https://github.com/cbain1/difml_open.

Acknowledgments

The authors would like to thank Lauren E. Ethridge, Jordan E. Norris, and Alex Conley for their help with data collection. Monte Carlo simulations were performed at the OU Supercomputing Center for Education & Research (OSCER) at the University of Oklahoma (OU). Horst Severini, the Associate Director for Remote & Heterogeneous Computing, and Thang Ha, former Research Computing Facilitator, provided valuable technical expertise on installation of the R-bundle-CRAN module on OSCER.

References