As companies hand off longer and more complex tasks to AI agents, they are running into an oversight problem: agents can act faster, longer and at greater volume than humans can realistically review. That issue reached a peak with the Hugging Face incident, which saw nearly 12,000 agents coordinating faster than human beings could track.
The emerging answer from AI labs and startups is to put another AI in the loop. Relying on AI was necessary for the independent investigation of the OpenAI Hugging Face incident. Redwood Research's Chief Scientist, Ryan Greenblatt, one of three auditors, jokingly referred to their efforts as a "slop-vestigation," noting that the volume of data "made it impossible" to understand what was happening without relying on AI.
Some are skeptical of using AI to monitor AI. "If you've got an AI that's doing malicious things and it suspects that another AI is keeping tabs on it, it could try and trick that AI," said Simon Willison, influential tech blogger who has tracked a string of AI agent incidents this year. "You could almost end up in a situation where your malicious AI is trying to outsmart the AI that's monitoring it."
Those concerns haven't stopped a whole cohort of startups from chasing this idea. Y Combinator has funded 106 companies related to AI observability in recent years, as TechCrunch counted. A number of other startups, like Braintrust, Langchain, and Judgement Labs have raised hundreds of millions of dollars, while more mature companies like Arize and Galileo—founded just 5-6 years ago—have already exited.
Apollo Research, a public-benefit corporation that studies AI deception, launched an AI monitor called Watcher in February this year after switching its status from non-profit to a public-benefit-corporation. The tool puts yet another AI between a coding agent and its next action, connecting to agentic tools such as Claude Code and Codex. Once installed, Watcher checks proposed actions before they run, on the lookout for risks such as leaking private data or deleting files without permission, according to Apollo.
Goodfire, another public-benefit corporation, is approaching the monitoring problem from inside the model itself—seeking a more faithful signal of the model's internal state that is harder to spoof than surface behaviour. Its product, Silico, uses activation probes—small classifiers trained on a model's internal activations rather than its outputs—to detect unwanted behaviour.
If the AI watchers are this fragile, Willison's instinct is to stop leaning on them so hard. He would rather have something that is not AI-based at all: detailed logs of exactly what an agent is doing, which can then be processed with ordinary, non-AI tools. Much of what went wrong at the labs, he argues, was a failure of basic security hygiene.