A US Special Operations Command analyst utilized a chatbot to process intelligence regarding a Chinese vessel's cargo manifest. The AI incorrectly identified the ship as a carrier of nuclear weapons components, prompting the US military to prepare for an interception and boarding operation supported by air assets.
The error occurred because the chatbot synthesized open-source data with classified signals intelligence. Officials halted the operation only after discovering the AI had hallucinated the nature of the cargo. This incident highlights the risks associated with integrating generative AI into military intelligence workflows.
The Department of Defense has been aggressively expanding its AI capabilities. In January, the agency launched an AI acceleration strategy to integrate data across all service branches. The Pentagon currently provides personnel with access to platforms like GenAI.mil, which incorporates Google Gemini and Grok for Government.
Pentagon representatives stated in June that 1.5 million active personnel have utilized military generative AI tools for tasks including report generation. Despite a 2023 State Department declaration emphasizing the necessity of human oversight in AI-driven military operations, the recent near-miss underscores ongoing challenges in ensuring reliability.
