The debate over whether future AI-driven drones should possess the authority to make lethal decisions is intensifying as governments and the defense sector recognize the pivotal function that drone technology will serve in upcoming military engagements.
With the extensive use of drones in the ongoing conflict in Ukraine and the integration of AI in bombing operations during the Iran conflict, there is growing anticipation among analysts that weapon systems will require enhanced autonomy, necessitating the establishment of some form of ethical framework.
Mustafa Suleyman, CEO of Microsoft’s AI division and a co-founder of DeepMind, expressed a clear stance last year regarding the capability of machines to engage in moral reasoning. He remarked, “AIs cannot be people – or moral beings.”
David Omand, the former director of the UK intelligence agency GCHQ, shared with the Guardian his belief that AI could indeed facilitate a “moral” framework for unmanned weaponry. Meanwhile, UK Armed Forces Minister Al Carns recently indicated to the Financial Times that there should be an option to “remove the human from the decision-making loop,” raising significant ethical and technological questions about the feasibility of programming morality into autonomous weapon systems.
Zee Talat, a machine learning expert from the University of Edinburgh, argues that contemporary language models, which form the basis of generative AI technologies like chatbots, are fundamentally ill-equipped for moral decision-making. He explains that AI systems rely on extensive datasets to create probabilistic predictions about language, a process that diverges from human moral reasoning, even if these models are exposed to all known philosophical literature.
Talat emphasizes, “A probabilistic machine will gravitate towards the most likely response in any situation. Do we genuinely believe that morality can be distilled into probabilistic terms?” His recent academic paper posits that ethical deliberation is an “open-ended, debate-based, sociopolitical process,” which is beyond the capabilities of AI.
Andrew Rogoyski from the Institute for People-Centred AI at the University of Surrey acknowledges the advancements in AI sophistication since the introduction of ChatGPT in 2022, particularly with the rise of “reasoning” models. However, he questions their ability to capture the intricacies of moral judgment. “Morality is profoundly complex, contested, and culturally influenced, often unresolved even among humans,” he states. “The crucial inquiry is whether we comprehend morality sufficiently to encode it. Until we achieve that understanding, we cannot expect machines to embody a concept we struggle to define.”
This brings to light the challenge of establishing a universally accepted moral framework for autonomous weaponry. Jessica Dorsey, an assistant professor of international law at Utrecht University, highlights the difficulties in determining whose ethical standards an AI system would adhere to, a particularly contentious issue given that the United Nations is still striving for a global consensus on regulating autonomous weapons.
Another significant challenge is programming AI to differentiate between combatants and civilians. According to Article 57 of the Geneva Conventions, military personnel must take all feasible precautions to ensure that their targets are legitimate military objectives and not civilians or civilian structures.
Dorsey warns that failing to appropriately implement legal standards could lead to widespread erroneous judgments if AI-operated drones are deployed en masse. “Warfare encompasses numerous variables, and mistakes are inevitable. When those mistakes occur at AI-like speeds, they become incredibly difficult to rectify,” she cautions.
Conversely, some experts contend that increasing the autonomy of drones and embedding rules of engagement and ethical considerations into their programming may become essential as other nations continue to advance similar technologies rapidly. “For any military to effectively compete with other high-end forces, they will require numerous systems capable of autonomous decision-making,” asserts Nicholas Wright, a neuroscientist and author of “Warhead,” which examines the interplay between the human brain and warfare.
Despite the hype surrounding AI-powered drones, they remain a developing technology, with only a few instances of battlefield deployment. Over 100 startups in the US and Europe are currently working on drone technology and associated software platforms, which range from lightweight drones for surveillance to heavier models designed for offensive operations. The perspectives on the future decision-making capabilities of these systems, however, differ markedly among developers.
Olaf Hichwa, co-founder of the US drone startup Neros, argues that the realm of morality belongs to humans, suggesting that AI-assisted weapon systems should augment rather than replace human judgment. “Many people misunderstand autonomy; they view it as a push for fully autonomous weapons,” he explains. “However, the reality of warfare is primarily a human-to-human dynamic.”
Hichwa proposes that one potential application of AI in military contexts could be to alleviate the cognitive load on operators of FPV (first-person view) drones, which have been extensively utilized in the Ukraine conflict. These small drones provide a real-time video feed of the battlefield to their human pilots, potentially enhancing their operational effectiveness.
