Synthetic Polling Method Raises Concerns Over Accuracy and Diversity

A growing trend in public opinion polling is synthetic polling, where companies build hundreds or thousands of artificial personas based on existing data to simulate real voters in focus groups and surveys. These AI personas are created using large language models (LLMs) that draw from demographic information and other relevant details.

Jennifer Stromer-Galley, a professor at the School of Information Studies, has expressed concerns about this method due to its failure to account for shifting public opinion. According to research, real voters often change their views on key issues over time as their circumstances evolve. This difference in behavior between real people and AI personas is a critical issue.

Synthetic polling has disrupted the traditional trustworthiness of public opinion polling, which has long been valued for its reliability and consistency. The use of chatbots to simulate voter opinions raises questions about how accurately they can capture the nuances of human decision-making.

The construction of AI personas relies heavily on background information and key demographic details relevant to the poll in question. Additionally, large language models can be fed with further materials or documents to flesh out the personas, creating a more detailed representation of voters and their potential views on various policies.

Polling vendors are exploring a new approach to gathering public opinion data using artificial personas created by large language models.

This method allows for the development of detailed policy stances and perspectives that can be used to simulate focus groups and interest-based surveys.

As researchers delve into this emerging field, they note that polling firms see synthetic personas as an attractive option due to their potential cost-effectiveness and reduced logistical burdens.

However, experts warn that relying on fixed instructions when building these personas may lead to a lack of diversity in opinions, failing to accurately reflect the views of the intended audience.

A significant concern in synthetic public opinion polling is not just the number of personas generated, but rather the bias within the AI models driving them. While vendors can adjust the demographic mix of their personas to match a target population, this does not address how the model answers for each profile.

For instance, consider a survey targeting rural Black voters in a small town. The synthetic approach may give typical answers based on the profile, effectively erasing actual human outliers who do not conform to the expected response patterns.

Adjusting only for demographics is insufficient, as it fails to capture the range of views that real people hold. This means that even if the number of personas matches the average demographics of the population, synthetic polling still struggles with outliers in survey responses.

The issue is further highlighted by a recent experiment conducted by the Pew Research Center. In this study, an AI model was tasked with taking on the personas of real members of its American Trends Panel and answering surveys alongside human panelists. The results showed that across nearly 300 questions, the AI results differed from the human results by an average of 12 percentage points.

The experiment's findings also revealed that nearly half of the questions had at least one answer choice that no AI respondent selected, raising concerns about how synthetic polling can accurately account for outliers and unexpected responses.

The concept of public opinion polling has undergone significant changes since its inception in the 1930s and 1940s.

Initially, scientific polling was seen as a way for people to understand their collective concerns and values, fostering a sense of belonging among citizens. However, this method has evolved over time, and today synthetic polling is being used to gather data on public opinion.

Synthetic polling relies on AI personas to simulate human responses, which can create a distorted view of the country's concerns and values. This method may lead to misunderstandings that are magnified, as Stromer-Galey suggests.

The use of synthetic voters instead of actual human respondents raises questions about how this changes our understanding of ourselves as a collective. Does relying on AI-generated data compromise the accuracy and representativeness of public opinion polling?

As we increasingly rely on synthetic polling, it is essential to consider what aspects of human experience are lost in translation when replacing real people with artificial ones.

The rise of synthetic polling has led to a new wave of AI personas that mimic human opinions, but research suggests these digital duplicates don't always behave like real voters when faced with changing circumstances.

While AI personas can be programmed to adapt their responses based on new information, they often fail to exhibit the same level of cognitive flexibility as humans, leading to inconsistent and sometimes unreliable results.