METR: Anthropic's AI Progress Has Roughly Doubled in a Year

Anthropic's models are improving about twice as fast as they were a year ago, according to a new report from the AI safety research group METR published on Thursday. The company is using AI to build more capable AI systems, and for now that approach shows no clear sign of slowing down.

The report is based on an 11-week review conducted inside Anthropic and assesses the company's AI research and development processes. Its findings amount to some of the clearest evidence so far that AI is accelerating its own development. What was withheld from the public version is proving about as contentious as what was disclosed.

Development speed has doubled

METR says benchmark scores suggest Anthropic has made roughly two years' worth of progress over the past year, measured against the historical rate. The researchers say they are confident the speedup does not come solely from throwing more compute at model training, which has long been one of the main drivers of AI progress.

According to the report, as of October most of the acceleration is attributable to AI's own contribution, and that contribution is growing over time. The conclusion rests on interviews with Anthropic employees and informal Slack surveys, which point to substantial productivity gains. Employees using the company's most capable internal model reported, by their own account, being around 12 times more productive on average than they would be without AI assistance. That is roughly double the level employees reported a few months earlier when using Mythos 5.

AI is still in a supporting role

METR says most of today's momentum comes from AI assisting human researchers rather than working independently. Still, the researchers say there are early signs that fully autonomous contributions could soon play a larger part. They also stress that they found no clear indication, such as diminishing returns or an observable bottleneck, that would stand in the way of an even faster acceleration.

Another notable point in the report is that Anthropic's researchers have access to more advanced models than those available to the public. METR says the company has finished training three unreleased models in the past six months. Codenamed Model 2, Model 3 and Model 4, they completed training in July, August-September and September-October respectively. METR also identified an unreleased model dating back to 2022, referred to as Unreleased 2022. The researchers note these are the codenames they were given and that the company may use different names internally.

Redactions raise transparency questions

Anthropic gave METR access to a wide range of internal material, including confidential research, model performance data and employee interviews. METR acknowledges that the company shared more than originally planned and exceeded its expectations. Access was not unlimited, however: some data requests were refused, and the report describes the flow of information as relatively narrow. The review also ran separately from the "employee-like" access program that was announced for safety researchers.

The central dispute stems from the redaction authority the agreement gave Anthropic. METR says it submitted the full report, roughly 100 pages, to the company, and that in the public version Anthropic removed much of the evidence METR considers most informative. According to the group, many of the redacted pieces of evidence can be reasonably interpreted in more than one way, which means the public must largely trust METR's judgment rather than weigh the uncertainties of the findings for itself. METR says Anthropic made the redactions on the grounds that publishing the information could benefit its competitors. METR says most of the redacted material is very high-level, but that it cannot comment on whether the commercial rationale is justified.

The findings arrive as Anthropic is reportedly preparing for an IPO and as its CEO, Dario Amodei, is calling on AI labs to hit the brakes on development.