Nvidia has published new research indicating that the 'harness' surrounding an AI model is more important than the underlying model for long-horizon tasks. The research demonstrated that by using a custom harness, Claude Opus 5 achieved a 100% score on the interactive reasoning benchmark ARC-AGI-3.
The custom harness, which was tweaked to handle memory effectively and included a 'supervisor' component, significantly improved performance. Without this harness, Claude Opus 5 scored 30%, which was the top result among all models tested on the benchmark.
Adel El Hallack, vice president of product in Nvidia’s AI unit, explained that an AI agent is more than just the model's API. It encompasses the model, the scaffolding (harness), and the runtime with associated skills and libraries. The harness is responsible for managing memory, context, and feedback, which are vital for tasks requiring many decisions over time.
The research also highlighted the importance of a supervising agent to guide the main agent, especially when it encounters difficulties or strays from its objective. Nvidia produces open components for building harnesses under its Nemo brand, with some being commercial and others openly available.