Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion

Overview

• Researchers investigated whether large language models (LLMs) can accurately estimate public opinion, using the example of vote choice in Germany.
• They generated a synthetic sample of personas matching characteristics of a German election survey and asked the LLM GPT-3.5 to predict each respondent's vote choice.
• The LLM's predictions did not match the actual survey data, exhibiting a bias towards the Green and Left parties and missing the complex factors influencing individual voter choices.

Plain English Explanation

Researchers wanted to see if large language models (LLMs) could be used to estimate public opinion, using voting behavior in Germany as an example. LLMs are AI systems trained on vast amounts of text data, which allow them to generate human-like responses.

The researchers created fictional people with characteristics matching a real German election survey. They then asked the LLM called GPT-3.5 to predict how each of these people would vote. The researchers compared the LLM's predictions to the actual survey data.

Unfortunately, the LLM's predictions did not match the real-world survey results very well. The LLM seemed to have a bias towards the Green and Left political parties, and it failed to capture the nuanced factors that influence how individual voters make their choices.

This study adds to the growing research on the conditions under which LLMs can be used to study public opinion. The findings suggest that LLMs may not accurately represent the full range of opinions in a population, and there are limitations to using them for this purpose.

Technical Explanation

The researchers generated a synthetic sample of personas matching the individual characteristics of respondents from the 2017 German Longitudinal Election Study. They then asked the LLM GPT-3.5 to predict each respondent's vote choice and compared these predictions to the survey-based estimates at both the aggregate and subgroup levels.

The results showed that GPT-3.5 did not accurately predict citizens' vote choice, exhibiting a bias towards the Green and Left parties. While the LLM was able to capture the tendencies of typical voter subgroups, such as partisans, it failed to account for the multifaceted factors that influence individual voter choices.

By examining the LLM-based prediction of voting behavior in a new context, this study contributes to the growing body of research about the conditions under which LLMs can be leveraged for studying public opinion. The findings point to disparities in opinion representation in LLMs and underscore the limitations in applying them for public opinion estimation.

Critical Analysis

The paper acknowledges several contextual factors that might affect the generalizability of the findings, such as the relationship between the target population and the LLM's training data. The researchers also note that their study only examined vote choice and that LLMs might perform better at estimating public opinion on other topics.

However, the paper does not address the potential biases and limitations inherent in the survey data used as a benchmark. Survey data can also fail to fully capture the complexity of public opinion, particularly on sensitive political topics. It would be valuable to further investigate the discrepancies between LLM predictions and survey results, considering the limitations of both approaches.

Additionally, the study focuses on a single LLM (GPT-3.5) and a specific context (German voting behavior). More research is needed to understand how different LLMs and various public opinion domains might influence the accuracy and reliability of LLM-based estimates.

Conclusion

This study suggests that large language models (LLMs) may not be able to accurately estimate public opinion, at least in the domain of voting behavior in Germany. The LLM GPT-3.5 exhibited biases and failed to capture the nuanced factors influencing individual voter choices.

The findings contribute to our understanding of the limitations of using LLMs for studying public opinion. While LLMs have the potential to complement traditional survey methods, this research highlights the need for caution and further investigation into the conditions under which LLMs can reliably represent the diversity of public opinion.