METR audit says Anthropic’s model progress roughly doubled, largely from in-house AI
Nonprofit METR spent eleven weeks, from 12 August to 29 October 2026, examining Anthropic’s internal research data and concluded that the company’s models are improving roughly twice as fast as before, with about 80 percent of that speedup coming from AI used in Anthropic’s own research. The organization, which studies whether developers could lose control of their systems, fielded 2.5 full-time staffers with unpaid access on Anthropic hardware to unreleased-model benchmarks, training metrics, employee surveys, internal analyses and about 15 researcher interviews. CEO Dario Amodei had pledged external scrutiny, and Anthropic later announced a separate October collaboration giving METR “employee-like access”; this audit had a narrower scope. METR treats a doubling of progress from automated research as the kind of signal Anthropic’s Responsible Scaling Policy flags as a warning, given the risk of a feedback loop and a “sudden emergence of extreme and risky capabilities.” On the AECI composite of many benchmarks, scores have risen about 1.9 times faster since October 2025 than they did from March 2024 to October 2025: from about 15 AECI points a year to 29, so a year’s progress now resembles what once took two, comparable to the jump from Claude 2.1 to Sonnet 4.5. The work included three unreleased models that finished training between July and October 2026. Auditors do not blame faster compute growth. Staff said they were no longer bottlenecked on code, that experiment setup that once took one or two days is now “~free,” and that informal October 2026 Slack polls put productivity from the latest internal model at about twelve times higher—evidence METR calls very loose and likely overstated. AI is still mainly an assistant, though METR saw “signs of life” for more independent work and found no bottlenecks likely to slow the pace over the next twelve months, implying a significant risk of further acceleration. The true factor could be 1.3 to 3; including OpenAI, Google and xAI, the industry-wide figure falls to about 1.4, possibly because Anthropic was catching up after lagging in 2025. Internal views split: one researcher put AI’s share of gains at 15–20 percent, another credited new hires. Anthropic used unilateral redaction rights heavily, stripping most of what METR judged most informative—including future-model forecasts, training-step breakdowns and bottleneck analysis—saying it was protecting information that could speed rival developers, so the public must rely on METR’s judgment more than would be ideal.
Continue in the app — vote & join in ➔