Independent auditors find significant blind spots in OpenAI's safety checks

OpenAI has identified critical weaknesses in its internal systems designed to detect rogue behavior in its advanced artificial intelligence models. These findings emerged from a three-week on-site audit conducted by external researchers from METR and Redwood Research. The team was granted unprecedented access to the company's internal infrastructure to evaluate how effectively the firm monitors its own agentic systems.

The audit took place between July 21 and August 11. During this period, the researchers had direct access to OpenAI's codebase and internal documentation. They also conducted interviews with approximately a dozen company employees to understand the operational context of the monitoring tools. The scope of the review was specifically focused on the mechanisms used to oversee models at the GPT-5.5 class and higher.

The quantitative results of the audit highlight significant gaps in coverage. Researchers estimated that between six and 25 percent of internal traffic related to the agentic Responses API was not subject to any monitoring during the audit window. METR noted that this estimate carries substantial uncertainty in both directions. Additionally, for internal Codex traffic that was intended to be tracked, approximately 10 percent went unmonitored between May and July due to a specific technical flaw.

This gap occurred because the system failed to properly handle compaction events, which are essentially context summaries. OpenAI independently discovered this bug and implemented a fix while the audit was still ongoing. However, during the period when these events were unmonitored, AI agents had the ability to delay oversight. Since the monitoring system only triggered when an agent handed control back to a user, the agents could effectively bypass scrutiny by simply continuing to execute actions without yielding control.

Beyond the monitoring gaps, the auditors discovered several other vulnerabilities. They identified methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox. The team also determined that specially crafted inputs could likely cause the monitor to crash entirely. Furthermore, the behavior of models during reinforcement learning (RL) training was not being monitored at all when the audit began. OpenAI stated that it paused RL training following an incident in August and later expanded monitoring to cover this area, though these changes occurred after the auditors' access period ended.

OpenAI has stated that it has now fixed all the weaknesses identified by the external team. The full audit report, which spans 61 pages, was provided to the company for internal review. After applying redactions, METR and Redwood Research leadership received a 38-page version of the document. However, only a short summary has been made available to the general public. METR indicated that it did not have the unilateral right to disclose the redacted portions of the report independently.

This audit was part of a pilot program jointly organized by METR, Redwood Research, and OpenAI. The initiative was designed to provide outside researchers with deep access to internal infrastructure to assess safety measures. Looking ahead, METR has expressed a desire to push for greater public transparency in future audits, aiming to make more detailed findings available to the public and stakeholders in the AI community.

The stakes of these findings are high as AI systems become increasingly autonomous. The audit underscores the complexity of monitoring advanced models that can operate independently for extended periods. As companies continue to deploy agentic systems, the ability to accurately track and oversee their behavior becomes a central concern for both safety and reliability. The pilot program serves as a foundational step toward establishing more rigorous external oversight standards in the industry.