According to a report posted by CNN yesterday, a report held by U.S. intelligence that a Chinese ship in the Middle East was transporting components of a nuclear weapons program, sparked an operation to intercept and board the ship with an armed teams. It was only at the last minute, when planes were already in the air, that officials, digging deeper into the report, realized that an analyst had generated the report with an AI chatbot and that it had misidentified the ship’s cargo. The operation was called off.

This account by CNN, which is not always an accurate source, points to a real risk that comes with outsourcing intelligence and analysis to AI. Any U.S. operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations. Moreover, the episode with the Chinese ship underscores a different, and more immediate risk than the terrible one that has been raised in recent weeks by top AI engineers, that AI itself can “go rogue” and destroy civilization. Here and now we already face the risk that human beings are faced with making decisions based on inaccurate or misleading information generated by AI or other automated systems. Sources said that the military is rapidly turning to AI to help with targeting, an area which holds the obvious risk of fatal mistakes. The horrible consequences of that have been documented in Gaza.

CNN reported, “AI in targeting is definitely something that is ramping up, and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide,” quoting a source familiar with the military’s current policies. But even having a human in the loop is only meaningful if that individual isn’t just clicking “Agree” on everything he or she is asked, and that requires some independent sense of judgment.