Apple and Google are facing a direct challenge from the city of San Francisco, which has formally ordered the tech giants to remove numerous 'nudify' applications from their respective app stores. These apps, which utilise artificial intelligence to digitally alter photographs and create non-consensual intimate images, are at the centre of a legal dispute that could have far-reaching implications for content moderation and platform accountability in the digital age.
San Francisco City Attorney David Chiu issued letters to both companies, asserting that they have been aware for an extended period that these apps violate California state law. The law, which criminalises any activity that 'knowingly facilitates' or 'recklessly aids or abets' the creation of non-consensual deepfake pornography, was further strengthened in 2025 by legislation allowing victims to pursue civil actions against third-party facilitators. Despite these well-established regulations, the city contends that Apple and Google have continued to host and financially benefit from these problematic programs.
Chiu's office highlighted that both companies have been on notice regarding their role in 'processing payments for illegal purchases for almost a year'. Reports from the Tech Transparency Project in January and April repeatedly warned Apple and Google about the presence of 'dozens of apps' in their stores that sold deepfake non-consensual intimate images (NCII) in exchange for payments processed by the firms. The April report specifically accused Google and Apple of intentionally 'steering' users towards such apps, characterising them as 'key participants in the spread of AI tools that can turn real people into sexualised images'.
The financial aspect of this issue is significant, with Chiu suggesting to Wired that Apple and Google have likely generated 'millions of pounds in fees' from applications offering these services. The letters from the City Attorney's office warn that both companies could face substantial civil penalties for violating state law and have requested a response within 28 days. This action follows a growing trend where deepfake pornography, once primarily a concern for public figures, has become accessible through these apps, making anyone with a publicly available photo a potential target.
For UK businesses and consumers, this development underscores the increasing scrutiny on platform responsibility for user-generated and AI-generated content. While the immediate legal action is in California, the precedent set could influence regulatory approaches globally. The UK's Information Commissioner's Office (ICO) has been actively monitoring AI developments, and the EU AI Act, though still in its implementation phase, aims to establish a comprehensive legal framework for AI, including provisions that could impact such applications. Experts are noting that this case highlights the urgent need for robust ethical guidelines and technical safeguards in AI development and deployment to protect individuals from digital harm.