ChatGPT, Claude, Grok, and Gemini are the main chatbots used in a recent study on election-related queries

A recent study conducted by Yale researchers has shed light on how people use AI chatbots to gather information about elections. The researchers asked nearly 2,500 adults in 35 states with Senate races to query four different chatbots - ChatGPT, Claude, Grok, and Gemini - for election-related information.

These participants were chosen because they already use chatbots regularly, typically on a weekly basis. They used the chatbots to ask about basic facts such as who was running for office, but also to seek recommendations on which candidate to vote for. In fact, 36% of users asked the chatbot to help them pick a candidate.

The majority of users relied on the chatbots for straightforward information, but those seeking guidance on which candidate to support often provided their priorities and asked the chatbot to evaluate candidates based on these criteria. This type of query is more prone to errors because it involves interpreting complex data about candidate positions, rather than simply retrieving basic facts.

The accuracy of a chatbot's recommendations can depend significantly on its sources. For instance, information from a candidate's official website may be considered more reliable than social media clips that are often taken out of context. The researchers likely took this into account when analyzing the results of their study.

It is worth noting that different chatbots recommended candidates at varying rates. While Grok frequently suggested candidates without being prompted to do so, ChatGPT was the most commonly used chatbot for election-related queries and recommended candidates more often than the other three options.

A significant number of users asked chatbots for subjective opinions on election candidates, highlighting a potential pitfall in relying on AI for decision-making support.

One notable example involved a user requesting the "pros and cons of each candidate in Virginia, a question that is inherently subjective and open to interpretation by the chatbot.

The results showed varying degrees of success among the different chatbots when it came to recommending candidates. When users asked who to vote for without providing any information about themselves, only one chatbot consistently offered a recommendation.

According to the study's findings, this chatbot, Grok, recommended a candidate in 14% of chats, while another popular option, ChatGPT, was more forthcoming with its recommendations once users shared their views.

In fact, ChatGPT recommended a candidate in 79% of chats after users had provided some information about themselves. Another chatbot, Claude, also made recommendations at a lower rate than ChatGPT but higher than Grok's initial rate.

The study also highlighted the disparity between the number of people using chatbots for general information and those specifically seeking election-related details.

The use of chatbots for election-related information has become increasingly common in recent years, but their effectiveness is still unclear. A study by the Brennan Center for Justice found that while chatbots were able to push back against certain falsehoods related to election disinformation campaigns, they also struggled with generating accurate and reliable information.

In particular, the study noted that AI models had difficulty distinguishing between fact and fiction when it came to election-related images and video. This was particularly concerning given the sophistication of deepfakes in recent years, which have become increasingly difficult to distinguish from real footage.

The study's findings were based on a series of tests conducted with several popular chatbots, including ChatGPT, Gemini, Grok, Meta AI, Runway, and Flux. The results suggested that many of these safeguards against misinformation were ineffective or inconsistently applied, which could have significant implications for civic engagement.

As the use of chatbots continues to grow, concerns are being raised about the impact on voter decision-making. With the complex and often opaque nature of AI training data sets, filters, and settings, it can be difficult to trust the accuracy of information provided by these tools. This has led some experts to warn that layering a chatbot onto an already dense political media landscape could further muddy the waters for voters.

The use of AI in this way is often referred to as cognitive offloading," where the burden of processing and evaluating complex information is shifted from the individual voter to the AI system. While this may seem like a convenient solution, it raises important questions about the potential risks and consequences of relying on these tools for election-related information.

The approach used by Yale researchers involved allowing participants to engage with AI chatbots naturally, without the constraints of standardized questions often employed in studies. This method allows for a more realistic assessment of how these tools influence users' decisions.

The distinction between this study and others that use pre-defined queries is significant, as it enables evaluations that take into account the actual questions users ask when seeking information from AI chatbots. According to the researchers, this approach represents a small but crucial step towards understanding the impact of AI on user behavior during elections.

Anthropic's analysis of 250,000 conversations with its Claude chatbot found that election-related queries were relatively rare, making up only 0.51% of all interactions. The company had expected a higher proportion of users to be asking about voting and elections in the lead-up to November 3.

When people did ask about elections, Anthropic's report noted that they often sought information that would help them make their own decisions, such as where candidates stood on specific issues. Requests that violated the company's election integrity policies were extremely rare, appearing in only 0.002% of conversations.

The next phase of this research will involve examining the accuracy and bias of AI models' responses to user queries, as well as what these models learn about users and whether their recommendations align with users' needs.

The chatbot's interactions with users played a crucial role in shaping its recommendations. Users often responded to the chatbot's questions, providing information that influenced its subsequent suggestions.

The researchers noted that the questions posed by the model itself can significantly impact what users share and what they are recommended. This dynamic exchange between user and model has significant implications for how AI systems present information.