Large Language Models Perform on Par with Experts Identifying Mental Health Factors in Adolescent Online Forums

Overview

• Mental health issues among children and adolescents have been on the rise in recent years.
• Large Language Models (LLMs) offer potential for cost-effective and scalable monitoring and intervention.
• Previous studies have not investigated LLM performance in this domain, particularly for open-ended information extraction.
• The researchers created a new dataset of Reddit posts from adolescents aged 12-19, annotated by expert psychiatrists for categories like trauma, precarity, condition, symptoms, suicidality, and treatment.
• They compared expert labels to annotations from two top-performing LLMs (GPT-3.5 and GPT-4) and also created synthetic datasets to assess LLM performance on generated data.

Plain English Explanation

Mental health problems in young people have been increasing over the past few years. New large language models (LLMs) like GPT-3 and GPT-4 could potentially help monitor and address these issues more efficiently and at a larger scale than humans alone.

However, previous research has not looked at how well these LLMs perform when it comes to understanding and categorizing mental health-related content, especially in open-ended situations where the possible responses are not pre-defined.

In this study, the researchers created a new dataset of Reddit posts made by adolescents aged 12-19. Expert psychiatrists labeled these posts for different mental health-related categories, like trauma, symptoms of disorders, suicidal thoughts, and treatment. The researchers then compared how well the LLMs GPT-3.5 and GPT-4 were able to match the expert labels.

They also created some synthetic (artificial) datasets to see if the LLMs performed better when the content was simpler and more straightforward, compared to the more complex real-world Reddit posts.

Technical Explanation

The researchers created a new dataset of Reddit posts from adolescents aged 12-19, which was annotated by expert psychiatrists for the following categories: TRAUMA, PRECARITY, CONDITION, SYMPTOMS, SUICIDALITY, and TREATMENT. They then compared the expert labels to annotations made by two top-performing LLMs: GPT-3.5 and GPT-4.

In addition, the researchers created two synthetic datasets to assess whether LLMs perform better when annotating data they generate themselves, as opposed to real-world social media posts. This was done to understand if the complexity of the real-world data was the primary challenge, or if there were inherent limitations in the LLMs' ability to handle certain concepts like negation and factuality.

The results showed that GPT-4 performed on par with human inter-annotator agreement on the real-world Reddit dataset. The performance on the synthetic datasets was substantially higher, suggesting that the complexity of the real-world data was a key factor in the LLMs' performance. However, the researchers found that the models still occasionally made mistakes in understanding negation and factuality, even on the simpler synthetic data.

Critical Analysis

While the findings suggest that LLMs like GPT-4 can achieve human-level performance on mental health-related text annotation tasks, the researchers acknowledge several limitations and areas for further research:

1. The dataset, while novel, is still relatively small, and may not capture the full diversity of mental health experiences among adolescents. Expanding the dataset size and scope could help validate the findings.

2. The synthetic datasets, while useful for isolating the impact of data complexity, may not fully reflect the nuances and contextual challenges of real-world social media posts. Further research is needed to understand the limitations of LLMs in handling such natural language data.

3. The study focused on a limited set of mental health-related categories. Expanding the scope to cover a wider range of issues, such as school bullying and eating disorders, could provide a more comprehensive evaluation of LLM capabilities in this domain.

4. The study did not explore the potential biases or ethical considerations of using LLMs for mental health monitoring and intervention. Ongoing research in this area should be considered to ensure the responsible development and deployment of such systems.

Conclusion

This study represents an important step in understanding the potential of Large Language Models to assist in addressing the growing mental health challenges faced by children and adolescents. The findings suggest that LLMs can achieve human-level performance on certain mental health-related text annotation tasks, but also highlight the need for further research to address the complexities and ethical considerations of using these models in real-world settings. As the use of AI systems in mental health care continues to evolve, it will be crucial to carefully evaluate their capabilities and limitations to ensure they are deployed in a safe and responsible manner.