Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis

Overview

• Researchers are investigating whether large language models (LLMs) can simulate human participants in experiments, opinion polls, and surveys.
• A key focus is mapping out the psychological profiles of LLMs by prompting them to respond to standardized questionnaires.
• However, translating LLMs' text responses into underlying personality traits is challenging.
• This study uses psychometrics, the science of psychological measurement, to evaluate the ability of GPT-3.5 and GPT-4 to assume different personas and respond to personality tests.

Plain English Explanation

Researchers are exploring whether large language models like GPT-3.5 and GPT-4 can be used to stand in for human participants in experiments, surveys, and opinion polls. A key part of this research is trying to map out the "psychological profiles" of these AI models - in other words, understanding their personalities and thought patterns.

This is challenging because translating an AI's text responses into underlying personality traits is not straightforward. To address this, the researchers in this study used psychometrics - the science of psychological measurement. They prompted the AI models to assume different personas, either generic or specific to real people, and then had them respond to standard personality tests.

The results showed that responses from GPT-4, but not GPT-3.5, using generic personas had reasonably good psychometric properties - meaning the AI's responses were similar to how real people would respond. However, when using specific demographic profiles, the responses from both AI models showed poor psychometric properties.

In other words, while GPT-4 may be able to somewhat realistically simulate generic human personas, it struggles to accurately reflect the individual-level behavior of specific people across multiple-choice personality tests. This casts doubt on the ability of current language models to truly simulate human-level behavior and decision-making.

Technical Explanation

This study investigated whether large language models (LLMs) such as GPT-3.5 and GPT-4 can be used to simulate human participants in experiments, opinion polls, and surveys. The researchers used psychometrics, the science of psychological measurement, to evaluate the ability of these LLMs to assume different personas and respond to standardized personality questionnaires.

The researchers prompted the LLMs to use two types of persona descriptions: generic (four or five random person descriptions) and specific (mostly demographics of actual humans from a large-scale dataset). They then had the LLMs respond to a range of standardized measures of personality constructs.

The results showed that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions exhibited promising, albeit not perfect, psychometric properties that were similar to human norms. However, the data from both LLMs when using specific demographic profiles showed poor psychometric properties.

These findings suggest that when LLMs are asked to simulate silicon personas, their responses are not reliable indicators of potentially underlying latent personality traits. The researchers conclude that this casts doubt on the ability of current LLMs to accurately simulate individual-level human behavior across multiple-choice question answering tasks.

Critical Analysis

The researchers acknowledge several limitations and caveats in their study. They note that mapping out the underlying personality traits of LLMs based on their text responses is an inherently challenging task, as the relationship between language and psychological constructs is complex and not fully understood.

Additionally, the researchers used a relatively small set of persona descriptions, both generic and specific, which may not fully capture the diversity of human experiences and behaviors. There is a need for further research using a wider range of persona profiles to better understand the capabilities and limitations of LLMs in this domain.

The researchers also highlight the fact that their findings are specific to the particular LLMs and personality measures used in the study. It is possible that other LLMs or different assessment tools may yield different results, and further research is needed to evaluate the generalizability of these findings.

Overall, this study raises important questions about the ability of current large language models to accurately simulate human-level behavior and decision-making, particularly at the individual level. While these models may show some promising psychometric properties when using generic persona descriptions, their performance deteriorates when tasked with simulating specific demographic profiles. This suggests that caution is warranted when considering the use of LLMs as a stand-in for human participants in experiments, surveys, and other research contexts.

Conclusion

This study investigated the ability of large language models (LLMs) like GPT-3.5 and GPT-4 to simulate human participants in experiments, opinion polls, and surveys by mapping out their psychological profiles. Using psychometric methods, the researchers found that while GPT-4 responses using generic persona descriptions showed some promising, albeit imperfect, psychometric properties, the responses from both LLMs using specific demographic profiles exhibited poor psychometric qualities.

These findings cast doubt on the ability of current LLMs to accurately simulate individual-level human behavior and decision-making across multiple-choice question answering tasks. While further research is needed to fully understand the capabilities and limitations of these models, this study suggests that caution is warranted when considering the use of LLMs as substitutes for human participants in various research contexts.