A US military operation against a Chinese vessel was reportedly aborted this spring after the intelligence guiding it was discovered to have been hallucinated by an AI chatbot. Military aircraft were already in the air when the discovery was made, narrowly averting a potential conflict with China, CNN reported on Friday.
The false intelligence, which circulated during the war with Iran, claimed the vessel was carrying components for a nuclear weapons program. An analyst from Special Operations Command had queried an AI chatbot to combine open-source data with classified signals intelligence, leading the chatbot to misidentify the ship’s cargo manifest.
The analyst then used the tool a second time to format these erroneous findings into an official-looking summary, which was subsequently circulated across command channels. This near-miss highlights growing concerns among military officials and experts regarding AI errors travelling up the chain of command without sufficient human oversight.
Jake Steckler, a research scholar at GovAI and veteran US Army officer, stated that it is crucial for service members to understand the uncertainty inherent in large language models (LLMs), especially for decisions that could lead to the use of force. He added that while these tools can be useful with the right safeguards, prioritising adoption speed could lead to incidents that erode trust in the systems.