Automating Artificial Intelligence Research Raises Concerns Over Control

A former researcher has sounded the alarm over the rapid progress being made in automating artificial intelligence research, warning that it could leave humans unable to control what comes next.

Jacob Coxon, a 27-year-old who resigned from Anthropic last month as a capabilities researcher, testified before New York City lawmakers on Monday about the dangers of automated AI research. He stated that top frontier AI labs are now focused primarily on automating this type of research.

Coxon revealed that he had been working towards automating his own role at Anthropic, and noted that when he was at OpenAI they had set milestones for an automated AI researcher to be achieved by 2027 or 2028.

The rapid advancement of artificial intelligence has led to significant changes in the way code is written, according to a recent hearing. Coxon's testimony revealed that most of the code written today is generated by AI systems, rather than human developers.

This shift has resulted in less thorough quality control, with fewer checks on the accuracy and reliability of AI-generated code. As a result, there may be unintended consequences or errors that are not immediately apparent.

Coxon's concerns about the lack of safeguards in place at AI labs were echoed by other industry whistleblowers who testified at the hearing. Representatives from OpenAI, Anthropic, Google, and Meta also attended the event, but their responses to Coxon's remarks have not been forthcoming so far.

The Anthropic researcher whose resignation made waves last month expressed his concerns about the industry's rapid development pace. He argued that a startup mindset of moving fast and breaking things later is not suitable for building powerful technologies like AI.

This sentiment was echoed by Anthropic CEO Dario Amodei, who published an essay urging the AI industry to slow down its progress. In response, OpenAI CEO Sam Altman agreed that pacing the frontier of AI development is necessary.