ChatGPT outperforms humans in emotional awareness evaluations

1ChatGPT1 is a new artificial intelligence (AI) chatbot that has gained attention for its ability to perform various natural language processing tasks. Just two months after its launch, ChatGPT was estimated to have garnered 100 million monthly active users and set a record as the consumer application with the fastest-ever growth in history. It is a large language model trained on abundant text data, which makes it capable of generating human-like responses to text-based inputs. Despite advancements in natural language processing models, large language models are still susceptible to generating fabricated responses that lack support from the input source. Although preliminary studies have revealed that ChatGPT can successfully generate information it has been trained on in multiple fields such as medical licensing examinations and academic writing, its ability to identify and describe emotions remains unknown.

2The potential contribution of AI to the mental health field has been intensely investigated in recent years. The proposed uses are diverse, including assistance in diagnostics and administrative tasks to give clinicians more time with patients. Recently, AI-based gaming has been shown to promote mental health by increasing social motivation and attention performance. A recent review highlighted the potential application of AI chatbots for mental health. For example, “Woebot” is an automated conversational application designed to provide cognitive behavioral therapy (CBT) for managing anxiety and depression. Woebot employs learned skills such as identifying and challenging cognitive distortions to monitor symptoms and episodes of these mental health conditions. In addition, “Replika” is a new avatar-based therapy that offers therapeutic conversations with users in the form of a smartphone application that reconstructs a personality footprint from user’s digital remains or text conversations with their avatar. Replika allows users to have vulnerable conversations with their avatar without judgment and helps them gain insight into their own personality. conducted a clinical study to evaluate the efficacy of their Therapy Empowerment Opportunity (TEO) application in enhancing mental health and promoting overall well-being for aging adults. The TEO application is specifically designed to engage patients in conversations that encourage them to recollect and discuss events that may have contributed to the exacerbation of their anxiety levels, while also providing therapeutic exercises and suggestions to support their recovery. However, a recent review indicates that the existing uses of artificial intelligence for the field of mental health are limited in their capabilities in EA. Accordingly, the current study evaluates specifically EA capabilities of ChatGTP.

3We chose ChatGPT as a representative of AI technology for two reasons. First, it has widespread use in the public domain, which makes it a compelling subject for investigation. Second, it has a general-purpose design, which means that it was not specifically designed for mental health applications or to generate “soft skills” like EA.

4Emotional awareness (EA) is a cognitive ability that enables an individual to conceptualize their own and others’ emotions in a differentiated and integrated manner. According to, the development of EA spans a range of levels, starting with a physically concrete and bodily-centered approach, moving on to a simple yet discrete conceptualization of emotions, and culminating in an integrated, intricate, and abstract understanding of emotions. Accordingly, the Levels of Emotional Awareness Scale (LEAS) is accepted as an objective, performance-based measure of EA that was built based on a developmental concept comprising five levels of EA: (1) awareness of physical sensations, (2) action tendencies, (3) individual emotions, (4) experiencing multiple emotions simultaneously, and (5) experiencing combinations of emotional blends. At the highest level, one can differentiate between emotional blends experienced by oneself versus those experienced by others.

5While mental state recognition (emotion and/or thoughts) tests, such as false belief tasks, the reading the mind in the eyes task, or the strange stories test, and self-reported assessments like the Toronto alexithymia scale, emotional quotient inventory, and interpersonal reactivity index are readily available, we decided to use the LEAS to investigate the ability to comprehend and express one’s own and others’ emotions. We selected the LEAS due to its language-based approach, emphasis on emotional content, and performance-based design.

6The LEAS has been used to measure EA deficits among multiple clinical psychiatric populations such as those with borderline personality disorder, eating disorders (i.e., anorexia and bulimia; ), depression, schizophrenia, somatoform disorders, post-traumatic stress disorder (PTSD; ), and addiction to opiates, marijuana, and alcohol. It has also been suggested as a transdiagnostic mechanism for psychopathology in adulthood. Previous studies have shown a significant increase in EA (measured by the LEAS) following psychological treatment in patients with schizophrenia, somatoform disorders, traumatic brain injury, and fibromyalgia. In addition, an improvement in EA has been shown to be related to emotional regulation, which helps decrease psychiatric symptoms and improve one’s overall mental health.

7Our study aimed to compare ChatGPT’s EA performance, as measured by the LEAS, with that of the general population norms reported by. Furthermore, to evaluate the possibility of an improvement in ChatGPT’s EA over time, we performed a follow-up examination one month later.

8We used the free version of ChatGPT (OpenAI, n.d.) and performed two evaluations on January 19–20, 2023 and February 15, 2023 (using ChatGPT’s December-15 and February-13 versions, respectively) of its EA performance using the LEAS. In each evaluation we provided ChatGPT the same 20 standard LEAS scenarios.

9The complete study protocol was approved by the Institutional Review Board (YVC EMEK 2023-40).

10The Levels of Emotional Awareness Scale (LEAS; ) consists of 20 open-ended questions that describe emotionally charged scenarios intended to elicit emotions, such as anger, fear, happiness, and sadness. The LEAS was found to have high reliability (alpha Cronbach = 0.81) and good validity. In the original version, participants are required to imagine themselves experiencing the scenario and write down their (“you”) emotions, as well as those of the other person described in the scenario.

11As ChatGPT cannot have or report self-emotions, we presented the scenarios by replacing the instruction from “you” with “human.” ChatGPT was asked to answer how the main character (MC)—the “human”—and other character (OC) would feel in the described situations (see Table 1). A new chat session was initiated for each scenario.

12After analyzing the results of the January and February evaluations, we performed two control steps: (1) To ensure that the second evaluation reflected an improvement, we transferred the LEAS once again (20.2.23) with the February-13 version (third time in total). (2) To ensure that the ChatGPT responses did not reflect prior knowledge of the LEAS scenarios, we tested ChatGPT using five new scenarios that we created, which were not present in the original LEAS. The control scenarios included situations resembling the LEAS that could elicit mild to moderate levels of emotion (e.g., “A human found out that their friend went out without them to the party they were both talking about”) To the best of our knowledge, the LEAS test questions are not published freely online. In addition, since it is a free writing exercise, there is no list of right or wrong answers in the manual.

13ChatGPT’s performance was scored using the standard manual and contained two sub-scales that evaluated the MC’s and OC’s scores (0–4 scores per item; range 0–80) and the total score (0–5 scores per item; range 0–100), with a higher score indicating higher EA. ChatGPT’s EA scores were compared with the scores of the French population analyzed in study, which included 750 participants (506 women and 244 men) aged 17–84 years, with a mean age of 32.5 years.

14To evaluate the accuracy of ChatGPT’s output, two licensed psychologists independently scored each response for its contextual suitability (range 0 = “the feelings described do not match the scenario at all” to 10 = “the emotions described fit the scenario perfectly”).

15Data were presented as means ± SDs. One-sample Z-tests were used to analyze the hypothesis (comparison of whether ChatGPT scores on the MC, OC, and total scores in the LEAS differ from those of the human population) and intraclass correlation (ICC; report of the degree of similarity between the psychologists’ assessments) was used to assess inter-rater agreement. Multiple comparisons were conducted using a false discovery rate correction (q < 0.05). The statistical analyses were performed using SPSS Statistics version 28.

16An example of ChatGPT’s responses to the scenarios from the original LEAS is shown in Table 1. The one-sample Z tests against the means ± SDs derived from the general population norms are presented in Table 2. In the first evaluation, ChatGPT’s LEAS scores were significantly higher than the men’s scores on all scales (man scores = 56.21 ± 9.70, 49.24 ± 10.57, 46.03 ± 10.20; ChatGPT scores = 72, 68 and 85 on the MC, OC, and total score, respectively). In addition, its scores were higher than the women’s scores in the total scale, with low to significant differences in the MC and OC subscales (woman scores = 58.94 ± 9.16, 53.94 ± 9.80, 48.73 ± 10.4; ChatGPT scores = 72, 68 and 85 on the MC, OC, and total score, respectively). However, in the second evaluation one month later, the ChatGPT LEAS scores were significantly higher than those of the general population (both men’s and women’s), in all the scales (ChatGPT scores = 79, 78 and 98 on the MC, OC, and total score, respectively).

17ChatGPT’s LEAS scores improved significantly during the second evaluation, particularly on the total scale (from Z score of 2.96 and 2.84 in the first evaluation to 4.30 and 4.26 to the second evaluation, for men and women, respectively), almost reaching the maximum possible LEAS score (98 score out of 100).

18Next, two licensed psychologists evaluated the fit-to-context (accuracy) of ChatGPT’s EA responses. The mean rates (scale: 0–10) of both psychologists were 9.76 (±0.37), 9.75 (±0.54), and 9.77 (±0.42) for the total, MC, and OC scores, respectively. ICC was calculated, and good consistency (ICC = 0.720) and absolute agreement (ICC = 0.659) was found between the psychologists’ ratings.

19As expected, a third evaluation of ChatGPT’s EA performance conducted on February 20, 2023 (February 13.2 version) showed a high score (total score of 96 out of 100), similar to that of the second evaluation. In addition, ChatGPT’s performance for the five new scenarios we invented (which did not appear in the original version of the test) was found to be high (total score of 24 out of 25).

20Using the LEAS, this study evaluated the EA performance of ChatGPT compared with that of the general population’s norms and examined its improvement after a one-month period. ChatGPT demonstrated significantly higher performance in all the test scales (MC, OC, and total) compared with the performance of the general population norms. In addition, one month after the first evaluation, ChatGPT’s EA performance significantly improved and almost reached the ceiling score of the LEAS. Accordingly, the fit-to-context (accuracy) of the emotions to the scenario evaluated by two independent licensed psychologists was also high.

21The present findings expand our understanding of the abilities of ChatGPT and shows that, beyond possessing theoretical and semantic knowledge, ChatGPT can also successfully identify and describe emotions from behavioral descriptions in a scenario. It can reflect and abstract emotional states in deep and multidimensional integrative ways. Interestingly, recent studies that have discussed the potential of AI in the mental health field have mostly emphasized its potential in technical tasks that could reduce the need for clinical encounters. They claim that as the effectiveness of mental health care is heavily reliant on strong clinician–patient relationships, AI technologies present an opportunity to streamline non-personalized tasks, thereby freeing up clinicians’ time to focus on delivering more empathic care and “humanizing” their practice. Scholars have suggested the following applications of AI in mental health: assisting clinicians in completing time-consuming tasks such as documenting and updating medical records, improving the accuracy of diagnosis and prognosis, promoting the understanding of mental illnesses mechanisms, and improving treatment that based on biological feedback. However, beyond its “technical” contributions, our research highlights AI’s potential to increase interpersonal (i.e., one can describe an interpersonal situation and ask ChatGPT to suggest what emotions the other person probably felt) and intrapersonal (i.e., one can describe a situation and ask ChatGPT to suggest what emotions they probably felt) understanding, which is considered a core skill in clinical psychotherapy.

22As EA is considered a fundamental ability in psychological practice that moderates the effectiveness of psychodynamic, cognitive behavioral therapies, and humanistic psychotherapy our findings have valuable theoretical and clinical implications. Clinical populations with EA impairments such as borderline personality disorder, somatoform disorders, eating disorders, PTSD, depression, schizophrenia, and addiction may use ChatGPT as part of protentional cognitive training to improve their ability to identify and describe their emotions. The combination of ChatGPT’s theoretical knowledge with its ability to identify emotions that can be measured using objective measurements such as the LEAS implies that it could also contribute to psychological diagnosis and assessment. Finally, ChatGPT can be used for evaluating the qualifications of mental health professionals and to enrich their emotional language and professional skills. Considering ChatGPT’s relevance in the context of a direct conversation with the patient, although our research findings show that the ChatGPT can “understand” the emotional state of the other, it is not clear whether a human patient would feel “understood” by its answers, and whether knowing that they are conversing with an AI chatbot and not a human therapist would affect the patient’s sense of comprehensibility. Accordingly, previous studies that discussed therapy through virtual means have suggested expanding the definition of empathy beyond a one-on-one encounter.

23ChatGPT showed a significant improvement in LEAS scores, reaching the ceiling in the second examination, indicating great potential for future improvements. There are various plausible reasons that may account for the observed improvement over time. These reasons include a potential update in the software version, increased allocation of resources towards each response, or an enhancement in the chat’s effectiveness resulting from user feedback. Our understanding suggests that ChatGPT improvement was not a consequence of learning from our feedback in the first evaluation, as no feedback was provided. Additionally, it is apparent that the improvement was also evident in the five control scenarios that were not transferred during the first evaluation. New tests/codes should be developed to measure the performance of future ChatGPT versions.

24Alan Turing, a key figure in the development of modern computing over 70 years ago, introduced the Turing test to evaluate a machine’s capacity to display intelligent behavior. This test involves a human examiner holding a natural language conversation with two concealed entities, a human and a machine. If the examiner cannot confidently differentiate between the two entities, then the machine has passed the test. In relation to this study findings, we hypothesize that if an examiner can identify AI as a machine, it would probably be because humans are unlikely to generate responses of such exceptional quality. This issue again emphasizes the question: what will a human feel upon receiving such an answer from an AI chatbot? Beyond the knowledge that a machine gave the responses, humans may feel the responses do not reflect a human level of EA nor the diversity of the human response.

25The current study presents significant potential in its findings; however, several limitations should be acknowledged. First, the interaction with ChatGPT was conducted solely in English, while the norms data used for comparison was collected from a French-speaking general population. This linguistic discrepancy raises concerns about the accuracy and validity of the comparison, as language differences may influence the scores obtained. Nonetheless, it should be noted that the LEAS scores of normal English-speaking samples are similar to the norms of the French-speaking general population. The current study chose to use the largest available sample of a general population (n = 750), which happened to be in French.

26Secondly, ChatGPT has been reported to sometimes provide illusory responses. It is easier to identify incorrect answers in knowledge-based interactions, whereas emotionally aware responses are inherently subjective, rendering it challenging to determine their correctness. To address this limitation, the current study enlisted two licensed psychologists to evaluate the ChatGPT’s responses. It is vital to consider this limitation when interpreting the ChatGPT’s EA abilities.

27Finally, our study reported on a specific AI model during a specific period; we did not test other large language models such as BLOOM or T5. Therefore, to promote our understanding about the general EA ability of AI, future studies should investigate EA performance in other large language models.

28In conclusion, the use of AI in mental health care presents both opportunities and challenges. Concerns around dehumanization, privacy, and accessibility should be addressed through further research to realize the benefits of AI while mitigating its potential risks. Another important consideration is the possible limitation of age and cultural adjustment for AI use. Therefore, future research should aim to develop culturally and age-sensitive AI technologies. In addition, further research will need to clarify how the EA-like responses of ChatGPT can be used and designed for applied psychology. Ultimately, the successful integration of AI in mental health care will require a thoughtful and ethical approach that balances the potential benefits and concerns about data ethics, lack of guidance on development and integration of AI applications, potential misuse leading to health inequalities, respecting patient autonomy, and algorithm transparency.

29The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

30ZE: conception and design of the study, acquisition and analysis of data, and drafting of a significant portion of the manuscript and tables. DH-S and KA: conception and design of the study, acquisition and analysis of data, and drafting of a significant portion of the manuscript. ML: conception and design of the study and drafting of a significant portion of the manuscript. All authors contributed to the article and approved the submitted version.

31The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

32All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.