Emotions, their expression, and understanding are often described as unique characteristics of human life and development; however, with the growing sophistication of computer vision and machine learning, computing technology is rapidly shrinking the disparity between human and artificial intelligence. This evolution is particularly marked by a redefinition of artificial intelligence (Lisetti & Schiano, 2000). While originally referring to computers’ ability to perform cognitive tasks, artificial intelligence has now expanded to include a variety of subfields, including artificial wisdom (Jeste et al., 2020), and emotional intelligence (Erol et al., 2020; Poria, Majumder, Mihalcea, & Hovy, 2019). These advancements represent the latest hurdles computing technology must jump to match human intelligence capabilities (Schuller & Schuller, 2018), which has the potential to enhance psychological research. The expansion of emotion detection software is highly relevant to the improvement of measurement in psychological research and practice.

1 Introduction to Emotion Recognition API

Application programming interface (API) is a broad term that describes any means of communication between two or more computer programs. In particular, emotion recognition APIs allow the synthesis of computer vision, machine learning algorithms, deep learning neural networks, and other components in order to accurately detect and label human emotions (Deshmukh & Jagtap, 2017). The emotion API performance is further enhanced by cloud-based support, which continuously supplies learning algorithms with severs full of facial and emotional data (Khanal, Barroso, Lopes, Sampaio, & Filipe, 2018). Naturally, technology giants with the largest cloud infrastructure (e.g., Amazon, Microsoft, Google), are the most equipped to construct accurate emotion recognition programs. While specific expressions that can be detected vary from program to program, most algorithms are minimally equipped to identify the six basic human emotions: disgust, contempt, anger, fear, surprise, and sadness (Deshmukh & Jagtap, 2017).

The leading iterations of this technology are Microsoft Azure and Google Cloud Vision, which offer distinct advantages over one another in emotion recognition (Khanal et al., 2018). Microsoft’s API triumphs in overall accuracy, reporting high true positive (TP) rates for straight-facing and partially-straight facing profiles (Half Left Face TP = 60%; Straight Face TP = 74.9%; Half Right TP = 57.4%). Google’s API, however, can detect a wider range of facial profiles, particularly side-facing, but with reduced accuracy (Full Left Face TP = 7.3%; Half Left TP = 42.9%; Straight TP = 45.2%; Half Right TP = 43.2%; Full Right TP = 10.4%). The lack of non-frontal facial recognition is a significant limitation, but the implementation of new machine learning frameworks is gradually improving detection accuracy (Lin, Ma, Gong, & Wang, 2022).

It is important to mention that programming limitations are often readily addressed in future software updates, but sampling limitations require more targeted attention. Emotion APIs are typically trained with large samples of facial and corresponding emotion data, but a lack of diverse data often makes it difficult to account for physiological differences in emotion expression among different groups (Hernandez et al., 2021). Due to convenience sampling, training samples predominantly consist of white, young adults in America. This produces significant racial and age bias effects, which further confound previous accuracy estimates. For example, Microsoft and Amazon’s APIs are more likely to label Black participants’ neutral faces as angry or contemptuous compared to white participants exhibiting the same emotion (Kyriakou, Kleanthous, Otterbacher, & Papadopoulos, 2020; Rhue, 2018). Additionally, these programs demonstrate reduced accuracy with middle aged and older adults compared to young adult participants (Kim, Bryant, Srikanth, & Howard, 2021). Gender bias used to be a serious concern in previous API iterations (Klare, Burge, Klontz, Vorder-Bruegge, & Jain, 2012), but current research suggests that this disparity was addressed in recent updates (Kim et al., 2021). Whether through sampling adjustment or algorithmic improvement (Howard, Zhang, & Horvitz, 2017), it is likely the emotion recognition API will become more accurate with respect to racial and age bias, but progress in this area requires selective attention to improving representation.

2 Current Applications of Emotion Recognition API

The present review searched electronic databases (i.e., Google Scholar and Psyinfo) using five term categories: “emotion API”, “emotion detection”, “psychology”, “intervention”, and “emotion deficit.” Studies published since 2017 were included in the review if they (a) were published in English, (b) developed a new intervention using emotion recognition API, and (c) targeted individuals with emotion-related deficits. Current publications on emotion recognition API have seldom reached the mainstream of psychology research, as most studies exploring this technology have focused more on the computer science and algorithmic strength of software than its applications in measuring psychological constructs. Nonetheless, key methodologies have emerged in clinical, neurodevelopmental, and educational psychology research (See Table 1).

In clinical and neurodevelopmental areas, emotion detection software has assisted with the monitoring, treatment, and education of various individuals with emotion-related deficits. Byrne, Bogue, Egan, and Lonergan (2016) writes that “psychological mindedness,” the process of identifying and describing emotions, is an “explicit mentalizing capacity that is needed to engage effectively in psychotherapy.” Many talk-therapy techniques rely upon a baseline level of emotional intelligence, requiring that individuals are able to understand their own and others’ emotions. However, clients with emotional deficits struggle with emotion recognition, and, therefore, may not benefit from talk-therapy. Thus, emotion-related interventions are an important gateway step to other substantive areas of mental health treatment.

An expansive body of literature has investigated interventions for improving psychological mindedness in neurodevelopmental disorders, particularly Autism Spectrum Disorder (ASD) and Attention Deficit-Hyperactivity Disorder (ADHD). These populations often struggle with reduced empathy (Baron-Cohen & Wheelwright, 2004; Da Fonseca, Seguier, Santos, Poinso, & Deruelle, 2009; Uekermann et al., 2010) and emotion self-regulation (Braaten & Rosen, 2000), producing significant behavioral problems (Milton, 2012). The prime window to treat emotional deficits is during early childhood, but many individuals are diagnosed later in life. As the brain matures and patterns of dysfunctional social cognition become fixed, it is incredibly difficult to teach fundamental skills like empathy (Baron-Cohen, 2009). Given the behavioral consequences of ASD and ADHD in adolescents and adults, it is important that current emotional-deficit interventions are expanded to include populations in late-stage treatment.

Emotion recognition API, thus, is valuable because it can be readily incorporated in a variety of intervention settings and stages, from diagnosis to late-stage treatment (Liu et al., 2017). Regarding diagnosis, Manfredonia et al. (2018) used facial expression analysis software to measure differences in emotion expression and replicated diagnoses for ASD participants, ranging from 9-years-old to 54-years-old. Similarly, Jiang et al. (2019) synthesized emotion recognition and eye-tracking software to achieve a diagnosis accuracy rate that was competitive with those by professional psychologists. Post diagnosis, emotion API has been used to provide engaging education for ASD participants to build emotion-related skills. For example, Bharatharaj et al. (2017) developed a semi-autonomous robot presented as a toy parrot, which used Oxford API to monitor emotion regulation and practice social interaction with ASD children. Alharbi and Huang (2020) designed computer games that reward ASD children for accurately matching facial expressions in order to train empathy and communication skills. Many other popular games have been adapted using emotion API and computer-assisted instruction (Grossard et al., 2017), which improves both the accessibility and entertainment of diagnostic and intervention strategies for children with neurodevelopmental disorders.

The concern of emotion regulation in children also emerges within educational psychology literature, with multiple studies demonstrating that students with better emotion regulation ability perform better in the classroom and have higher levels of academic achievement (Gumora & Arsenio, 2002; Howse, Calkins, Anastopoulos, Keane, & Shelton, 2003). Naturally, students with emotional deficits, such as ASD, report much lower academic achievement rates than typically developing students (Ashburner, Ziviani, & Rodger, 2010). In E-Learning environments, emotion recognition API has been used to detect emotion changes in students with ASD during assessments (Chu et al., 2017), for targeted intervention strategies. This intervention was followed up by (Chu et al., 2020), which designed an emotion API-based intervention that utilized computer adaptive testing to identify and address learning stress in students with ASD. The result of this intervention significantly improved students’ math performance compared with baseline scores.

From a measurement perspective, some studies have raised concern about the reliability of the software’s emotion estimates. Borsos et al. (2022) evaluated the test-retest reliability of emotion API and found small but significant differences in the ratings. Flynn et al. (2020) observed group differences in the accuracy of emotion estimates between children and adults. However, it is important to note that both of these studies used emotion detection software (FR8 and iMotions respectively) that is meagerly discussed in the literature compared to the API produced by tech giants (e.g., Google Cloud and Microsoft Azure). These limitations are likely not representative of the method as a whole because these studies are operating on less-than-standard measurement tools. Nonetheless, inconsistency in emotion API responses are to be expected to some extent, which highlights the imperfect nature of emotion estimates. However, the adaptability of emotion detection software is a critical strength of this measurement approach, and as the software is incrementally improved over time, the accuracy of emotion estimates will also improve.

3 Potential Applications of Emotion Recognition API

Beyond neurodevelopmental disorders, clinical literature expresses a need for interventions to address a wide-range of psychopathology exhibiting emotion-related deficits. Alexithymia and empathy-related concerns are present in many other disorder classifications, particularly personality disorders (De Panfilis, Ossola, Tonna, Catania, & Marchesi, 2015; Thoma, Friedmann, & Suchan, 2013), and often lead to interpersonal dysfunction (Cook, Brewer, Shah, & Bird, 2013; Vanheule, Desmet, Meganck, & Bogaerts, 2007), internalizing and externalizing behavior (Aldao et al., 2016). Current personality pathology interventions often rely on self-report instruments, which have various validity concerns (Haeffel & Howard, 2010). Thus, the increased availability of emotion detection software has the potential to expand the range of options in how emotion-related experiments are designed. Emotion API has demonstrated its effectiveness in predicting Big Five personality traits and risk-taking behavior (Gloor et al., 2022), which is a significant facet of pathological personality (Watson & Clark, 2020). Detection software could be readily incorporated into studies interested in examining operational ways of measuring emotion dysregulation and psychopathological traits.

Regarding interventions, API strategies for neurodevelopmental disorders have not been tested on other psychopathology, but these interventions could generalize well with disorders that exhibit similar transdiagnostic traits. For example, Antisocial Personality Disorder and Narcissistic Personality Disorders often overlap with ASD and ADHD (Matthies & Philipsen, 2016). Emotion API could be an incredibly valuable tool in the measurement and design of pathological personality interventions beyond the scope of its current self-report methodology, which could benefit researchers and practitioners alike.

Emotion API interventions could also generalize to the broader, industrial-organizational need for better emotional intelligence trainings. Emotional intelligence is frequently measured in industrial-organizational contexts and is associated with multiple occupational outcomes, including job performance, retention, and interpersonal relations (Prentice, Lopes, & Wang, 2020). Thus, many industries declare a strong vested interest in screening for candidates with high emotional intelligence, or enhancing the emotional intelligence of their current employees. Facial expression is often described as a basic facet of emotional intelligence (Hildebrandt, Sommer, Schacht, & Wilhelm, 2015), and is often a targeted topic in emotional intelligence training programs. Employers and industrial-organizational researchers could capitalize off the automated and adaptive features of emotion recognition API to quickly improve employees’ emotional intelligence ability. API-based programs in emotion regulation could be inserted as a complement to existing modules on effective nonverbal communication and empathy.

As mentioned previously, emotion regulation is a critical component of students’ success in the classroom (Gumora & Arsenio, 2002; Howse et al., 2003), but other aspects of emotional functioning are relevant as well. Despite a common avoidance to express negative emotions, literature shows that negative emotions are a way to elicit support and build stronger relationships (Graham et al., 2008). Students who less openly express their emotions are less likely to receive help when struggling because they are often unable to call attention to signs of distress. That said, similar emotion expression interventions to the ones currently used for ASD could be helpful to acclimate these types of students to the importance of emotional intelligence. Alternatively, emotion API could be integrated into research focusing on instructors. Literature suggests that the emotion regulation ability of instructors also impacts student engagement and success in the classroom (Sutton et al., 2009; Wang & Ye, 2021). Detection software could complement classroom observation studies, generating ecological momentary assessments of instructors and their emotion regulation ability over the course of a lecture, which may be more accurate and reliable than current self-report or interview assessment strategies.

Broadly speaking, the integration of computational research methods would greatly benefit all areas of psychology, and this can especially be seen with emotion recognition API. The software allows researchers to easily collect and assign quantitative values to emotion-related data (Yannakakis, Cowie, & Busso, 2021), which increases the feasibility of collecting larger data without compromising quality of data. Emotion recognition API triumphs in efficiency over traditional measurement attempts, which are often long, unreliable, and cumbersome. Neurocognitive research could especially benefit from an increased efficiency in data collection, which is a contributing factor to concerns of low statistical power in current research (Button et al., 2013). An upgrade in statistical power is highly important and has the potential to increase the frequency and reproducibility of emotion-related research in clinical trials and neurocognitive work.

4 Conclusion

Although the integration of emotion recognition API is very much in its infancy in psychology, several subfields would benefit from an expansion of this highly adaptive area of measurement. Clinical research could enhance current intervention, develop new models of treatment, and establish new methods of measuring emotional functioning domains. Industrial-organizational research could develop new emotional intelligence indexes and training programs. Educational research could identify new ways of identifying and supporting students in the classroom. And neurocognitive research could generate more power and enhance the precision of neural mechanisms behind emotional expression. As this technology becomes more accessible, future studies should investigate API in all of these important disciplines and other, unidentified yet equally important areas. Although there are significant concerns of reliability and bias in the software currently, the incremental improvement of cloud-based programs confidently suggests that API is becoming a more reliable tool. Understanding emotions is a fundamental facet of human life experiences and emotion recognition API will allow psychologists to understand this phenomenon even further.

Acknowledgement

This study is supported by a grant from the Department of Education (R305D210023). However, the contents of the study do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.

References