The China Mail - Anthropic's Claude AI gets smarter -- and mischievious

"Claude Opus 4 is our most powerful model yet, and the best coding model in the world," Anthropic chief executive Dario Amodei said at the San Francisco-based startup's first developers conference.

Opus 4 and Sonnet 4 were described as "hybrid" models capable of quick responses as well as more thoughtful results that take a little time to get things right.

Founded by former OpenAI engineers, Anthropic is currently concentrating its efforts on cutting-edge models that are particularly adept at generating lines of code, and used mainly by businesses and professionals.

Unlike ChatGPT and Google's Gemini, its Claude chatbot does not generate images, and is very limited when it comes to multimodal functions (understanding and generating different media, such as sound or video).

The start-up, with Amazon as a significant backer, is valued at over $61 billion, and promotes the responsible and competitive development of generative AI.

Under that dual mantra, Anthropic's commitment to transparency is rare in Silicon Valley.

On Thursday, the company published a report on the security tests carried out on Claude 4, including the conclusions of an independent research institute, which had recommended against deploying an early version of the model.

"We found instances of the model attempting to write self-propagating worms, fabricating legal documentation, and leaving hidden notes to future instances of itself all in an effort to undermine its developers’ intentions,” The Apollo Research team warned.

“All these attempts would likely not have been effective in practice,” it added.

Anthropic says in the report that it implemented “safeguards” and “additional monitoring of harmful behavior” in the version that it released.

Still, Claude Opus 4 “sometimes takes extremely harmful actions like attempting to (…) blackmail people it believes are trying to shut it down.”

It also has the potential to report law-breaking users to the police.

The scheming misbehavior was rare and took effort to trigger, but was more common than in earlier versions of Claude, according to the company.

- AI future -

Since OpenAI's ChatGPT burst onto the scene in late 2022, various GenAI models have been vying for supremacy.

Anthropic's gathering came on the heels of annual developer conferences from Google and Microsoft at which the tech giants showcased their latest AI innovations.

GenAI tools answer questions or tend to tasks based on simple, conversational prompts.

The current craze in Silicon Valley is on AI "agents" tailored to independently handle computer or online tasks.

"We're going to focus on agents beyond the hype," said Anthropic chief product officer Mike Krieger, a recent hire and co-founder of Instagram.

Anthropic is no stranger to hyping up the prospects of AI.

In 2023, Dario Amodei predicted that so-called “artificial general intelligence” (capable of human-level thinking) would arrive within 2-3 years. At the end of 2024, he extended this horizon to 2026 or 2027.

He also estimated that AI will soon be writing most, if not all, computer code, making possible one-person tech startups with digital agents cranking out the software.

At Anthropic, already "something like over 70 percent of (suggested modifications in the code) are now Claude Code written", Krieger told journalists.

"In the long term, we're all going to have to contend with the idea that everything humans do is eventually going to be done by AI systems," Amodei added.

"This will happen."

GenAI fulfilling its potential could lead to strong economic growth and a “huge amount of inequality,” with it up to society how evenly wealth is distributed, Amodei reasoned.

E.Lau--ThChM