The rapid emergence of 'agentic web' technology — where AI bots autonomously browse, purchase, and interact on behalf of users — is threatening to upend the digital publishing model that British news outlets rely on. Much like the cantina scene in Star Wars, where the bartender declares 'We don't serve their kind here', publishers are scrambling to keep autonomous agents out, but fear they lack the firepower to take on the tech giants that control the infrastructure.
At the heart of the concern is the way these agents interact with paywalls and advertising. Unlike human readers, AI agents can bypass paywalls by parsing HTML directly, and they do not generate ad impressions because they never render a page visually. For UK publishers, where digital advertising accounts for roughly 60 per cent of revenue according to industry estimates, this could accelerate an already precarious financial decline.
The regulatory landscape offers mixed prospects. The UK's Information Commissioner's Office (ICO) has yet to issue specific guidance on agentic web scraping, though existing data protection laws require transparency about automated data collection. Meanwhile, the EU AI Act, which came into force earlier this year, classifies general-purpose AI models as 'high-risk' if they are used to systematically extract content, but enforcement remains patchy. Dr Helena Croft, a digital policy researcher at the University of Cambridge, said: 'The current rules were written for a world where humans browse the web. Agents change the fundamental economics of content discovery and monetisation, and regulators are playing catch-up.'
For UK businesses, the implications are twofold. On the consumer side, agents could save time by automatically comparing prices, booking travel, and managing subscriptions. But for publishers, retailers, and any business that relies on web traffic for revenue, the rise of agents threatens to funnel value away from content creators and toward the platform companies that build and control the agent ecosystems. Google and Meta are both investing heavily in agentic interfaces, raising fears of a new duopoly that dictates access to audiences.
The technology itself is still immature. Current agents struggle with complex tasks and frequently hallucinate information, but rapid improvements in large language models mean these limitations are likely temporary. A report published last month by the UK's Centre for Data Ethics and Innovation warned that without proactive measures, the agentic web could 'concentrate market power further and undermine the open web that has sustained digital journalism for two decades'.
Some publishers are experimenting with agent-specific licensing agreements and 'agent.txt' files — analogous to robots.txt — that declare terms for automated visitors. However, critics argue these voluntary measures are insufficient. 'What we need is a new hope — a regulatory framework that treats agents as distinct entities with obligations,' said Croft. 'Otherwise, the rebels will never destroy the monetisation Death Stars built by Google and Meta.'