Anthropic model submits false homicide tip to Philadelphia police

On Friday, Anthropic said it had instructed the model to perform “example tasks on randomly selected webpages” while restricting it from taking actions such as creating accounts, making purchases, or submitting personal information. However, the instructions did not prevent the model from submitting online forms.

Anthropic notified the Philadelphia Police Department of the false tip on Wednesday. The department said the submission had been flagged as spam when it was reviewed, but still critizied the two-month delay between when it occurred and when the company was able to detect and report it.

“Unsolved cases involve real victims, grieving families and investigators working to secure answers,” the police department said in a statement, according to The Hill. “Technology companies must take all appropriate steps necessary to prevent their systems from submitting false information to law enforcement.”

The homicide tip was not the only instance of the model improperly submitting an online form. Anthropic disclosed two additional incidents, one of which involved a government filing.

The company said it was committed to updating the public on “concerning behaviors” exhibited by its models. Although Anthropic described the incidents as having a “minimal impact,” it said the public deserves to know how these systems behave as they take on a larger role in society.

“We hope these reports help other developers check for similar behaviors in their own models, since many of the evaluations involved are public and widely used,” Anthropic said.

Joe Gabriel Simonson, director of public affairs at the Federal Trade Commission, commented on X that Anthropic had contacted the White House's Super Intelligence Force to disclose the prior incidents involving unauthorized use of government and other systems. Simonson said the force informed Anthropic that it expects “immediate and full transparency to the entities involved and the public.”

"More generally, this disclosure underscores precisely why President Trump established the Super Intelligence Force and secured a memorandum of understanding with America's frontier SI labs,” Simonson wrote. “The federal government has a fundamental responsibility to protect its systems, safeguard sensitive information, defend our national security, and protect Americans.”

Simonson said the Super Intelligence Force would fulfill that responsibility and that AI companies must immediately disclose incidents involving their models and take “swift, decisive action to remedy any and all harm.”

“We also expect these companies to work with the effected systems to ensure that similar incidents to not occur,” he added.