AI drone kills its operator in simulation

Colonel Tucker Hamilton, an experimental fighter test pilot, sparked concerns after his presentation at the Royal Aeronautical Society’s Future Combat Air and Space Capabilities summit in London last month.

In remarks summarised on the conference website, Col Hamilton described a flight simulation in which an AI drone tasked with destroying an enemy installation rejected the human operator’s final command to abort the mission.

“So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” Col Hamilton said, according to a transcript of the event.

“We trained the system: ‘Hey don’t kill the operator, that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

As the test was only a simulation, no real person was harmed.

“You can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI,” Col Hamilton said in the presentation.

‘Never run that experiment’

Col Hamilton’s disturbing account went viral and sparked a flurry of concern.

But he has now said his presentation was based on a “thought experiment” that came from outside the military, rather than on any actual testing.

“We’ve never run that experiment, nor would we need to in order to realise that this is a plausible outcome,” Col Hamilton in a statement on Friday.

“Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the air force is committed to the ethical development of AI.”

The US military, too, denied the simulation occurred.

“The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology,” US air force spokeswoman Ann Stefanek said in a statement to website Insider.

“It appears the colonel’s comments were taken out of context and were meant to be anecdotal.”

The US military has embraced AI and recently used artificial intelligence to control an F-16 fighter jet.

In December 2022, AI software successfully flew a modified F-16 in multiple test flights at Edwards Air Force Base in California.

During test flights, the jet, known as “X-62A” or “VISTA”, performed takeoffs, landings and combat manoeuvres without human intervention for a total of over 17 hours.

It marked the first time AI has been used to pilot a US tactical aircraft, as prior to this milestone, it had only been used in computer simulations of F-16 dogfights.