Independent AI Evaluators: The Anthropic-METR Case

Who checks, from outside the company itself, that an AI model is safe before it reaches millions of users? Since October 15, 2026, Anthropic has partly answered that question: the company gave a team of six independent evaluators from the nonprofit METR employee-like access to its models, internal tools, and San Francisco offices. For an SMB leader who already entrusts data and processes to Claude or a competitor, this announcement raises a simple question: can an AI vendor's safety claims be verified from the outside?

In short

• On October 15, 2026, Anthropic granted employee-like access to six evaluators from METR (Model Evaluation & Threat Research), a Berkeley-based nonprofit.
• METR can publish its analyses without Anthropic's approval, which can only require limited redactions (customer data, commercially sensitive information, model architecture).
• This is the top tier of a three-tier system of external evaluators Anthropic is building, with SecureBio and Accenture at lower access levels.
• Observers (San Francisco Standard, Palisade Research) note that METR's ties to major AI labs remain close, raising questions about how independent the evaluation really is.
• For an SMB, this is a new criterion to add to its AI vendor checklist: whether an outside party audits the vendor's safety claims at all.

Who is METR, and what did it just get at Anthropic?

METR (Model Evaluation & Threat Research) is a nonprofit founded in late 2023 in Berkeley by Beth Barnes, a former researcher at OpenAI and DeepMind. Its mission: evaluate the capabilities and risks of frontier AI models on behalf of labs and, increasingly, the public.

Until now, METR mostly worked by reviewing the risk reports labs publish themselves, somewhat like an outside auditor reviewing accounts the company already prepared. The access announced on October 15, 2026 changes the scale: six METR staff now have workspace inside Anthropic's offices, the same hardware as internal risk teams, and comparable internal tools.

A key point for judging how credible this setup is: Anthropic cannot veto METR's findings before publication. The company can only request redactions within predefined, narrow categories: customer data, commercially sensitive information, and architecture details that could enable misuse. METR, in turn, can publicly flag disagreement if it believes a redaction request goes too far.

Why this announcement comes now

The timing reflects a chain of internal and external pressure on AI labs.

This context matters because no US federal law currently governs frontier AI model safety specifically. Absent binding regulation, labs move forward through voluntary commitments, under pressure from public opinion, local authorities, and their own safety researchers. OpenAI has said it is finalizing its own contracts with outside evaluators.

The limits of this self-regulation

Honesty requires nuance here rather than celebrating the announcement uncritically.

Jeffrey Ladish of Palisade Research, a former Anthropic employee, sums up the difficulty: it's very hard to find AI safety talent with no ties at all to Berkeley or to the AI companies themselves. Other observers, such as researcher Brian Patrick Green of Santa Clara University, propose going further with an oversight board bringing together several independently funded organizations, plus rules limiting staff movement between labs and evaluators.

One more point worth noting: this month, OpenAI reportedly fired three safety researchers for allegedly sharing sensitive information with an unnamed outside group, an episode that illustrates the still-unresolved tensions around the role of these third-party evaluators.

What this means for SMBs choosing an AI vendor

An SMB doesn't negotiate directly with METR or with California regulators. But this news gives concrete questions to ask before signing, or renewing, a contract with an AI vendor.

This table illustrates a simple point: "having an outside evaluator" doesn't mean the same thing depending on the access level and publication rights granted. An SMB evaluating an AI vendor, directly or through its integrator, should ask for this level of detail rather than settling for a vague "audited by a third party" line on a marketing page.

FAQ

What is METR?

METR (Model Evaluation & Threat Research) is a Berkeley-based nonprofit, founded in late 2023, specializing in evaluating the capabilities and risks of frontier AI models on behalf of labs and the public.

Can Anthropic block publication of an unfavorable METR report?

No, not directly. Anthropic has no veto power over METR's conclusions. The company can only request redactions within predefined, narrow categories, such as customer data or certain model architecture details.

Is this kind of independent audit legally required?

No. In the US, no federal law currently mandates this kind of outside evaluation for frontier AI models. It is a voluntary commitment by Anthropic, made amid growing political and public pressure, particularly in California.

Should an SMB demand this kind of audit from its AI vendor?

It's not mandatory, but it's a relevant criterion to add to a vendor checklist, alongside data security and regulatory compliance. Asking whether an outside audit exists, and what publication power it carries, helps assess how serious a vendor really is.

Going further

An AI vendor's safety governance is just one criterion among others before building your processes on an AI tool. To structure that choice more broadly, see our guide to choosing your AI model and discover how other SMBs built their AI strategy in our customer success stories.