34:51
Kyiv, March the 12th, 2022. It's day 17 of the Ukraine war. A drone goes down between two rows of houses. With a wingspan of just over 1 m and bearing no visible weapon systems, it appears inconspicuous. It's crash is almost lost in the horror of the Russian invasion. And yet, it is of great significance. That's because it is a Kub-BLA. A suicide drone that can be controlled by artificial intelligence. It circles autonomously above the target area. Ready to swoop down at the right moment.
It is the harbinger of a technological quantum leap that will change war forever. Three, two, one. Launch, launch, launch. Military operations launched in seconds. Autonomously calculated drone strikes. AI deployed in the air, at sea, and on land. What once sounded like science fiction will soon be reality. AI is not only changing the way we drive and the way we generate text, it is also transforming the weapons we use to make war.
Despite international efforts to regulate them, the rise of autonomous weapons seems unstoppable. Newly sparked conflicts are pouring money into the arms industry. Tech-minded thinkers have been warning us for years about a looming loss of control. The danger of AI is much greater than the danger of nuclear warheads. By a lot. Could AI create a situation where a drone can select a target itself? I think we shouldn't allow that. Well, can it be done?
Sure. Thanks. The question is how you harness it. He who harnesses the AI will win. It is extremely powerful and it's going to have to be dealt with. And it is not only in war that we will encounter these systems. Pandemics, civil unrest, and crumbling democracies are the breeding grounds for technologies that we've only known in our darkest fictional dystopias. Artificial intelligence feels like another escalation of militarization within US police departments. We already have them functioning like occupying armies and they've taken that role very seriously.
Machines are supposed to solve conflicts that humans cannot. Experts are fiercely debating where this will lead us. We have the power to create these weapon systems as well as systems that can do something completely different. Addressing climate change, for example. It would be irresponsible not to develop it. This technology can be much more precise and avoid civilian casualties. And I'm not saying that we're already there, but in some cases it could make your warfare more humane. AI is supposed to make weapons systems faster, more precise, and more efficient. The price is less control. It's a trade-off governments are well aware of. But none of them want to fall behind. Because the cards on the international stage are being reshuffled.
France, in particular, is pushing for an independent European security architecture. It's also home to a highly productive defense industry. Nexter is a French defense group that manufactures a wide range of modern military technology. Nexter's robotics division is dedicated to the production of robots for the army, firefighters, and special police units. In addition to spy robots, Nexter also builds larger robotic vehicles like the armed Optio X20. It's not an easy task as the control of land-based robots poses great challenges.
Modern conflicts show that jammers offer one of the most efficient countermeasures against unmanned weapon systems. To ensure their autonomy, future machines will be designed to recognize landmarks, buildings, roads, and distances navigating just as humans do. Despite the enormous progress, the functionality of large land robot systems is still limited and mistakes could be fatal. But even skeptics see the signs of the times. New conflicts are creating new realities for which Europeans are ill-prepared. Marcus Reisner is a colonel in the Austrian armed forces.
His precise analysis of the Ukraine war have made him a sought-after expert. He has been deployed to numerous crisis regions around the world. And he's gained valuable experience with unmanned systems at the German drone company EMT. In 2018, he published a book on robotic warfare. Spurred on by the promise of increasingly efficient weapon systems, more and more countries are striving to automate their armies. Amid this new scramble, world powers are taking their claim to leadership all the more seriously. And the world's leader in AI development is still the USA. For several years, Jane Pinelis was the head of the artificial intelligence
testing division at the US Department of Defense. She also works at the Johns Hopkins University Lab for Applied Physics, a research institution with close ties to the Pentagon. So, AI, when you're talking about how it's used today, it really depends on how you define it, right? We can make an argument that if a system that just uses AI for some part of its functioning, that's an autonomous system, uh then I would argue that AI has been used in the DoD for 70 years or so, right? Like a torpedo, when you say go here and then it autonomously figures out how to get there. Where we are today, where we're pretty comfortable, is machines recognizing patterns in data. A machine picking out what a truck looks like and
being able to point to trucks in various scenarios or other types of vehicles. But, we're already moving into the area where machines can select and perform appropriate behaviors. If you look at the Israeli Harpy drone, for instance, the human points it in a direction, it says go to this area, uh and then within that area, it is capable of finding enemy radar and then targeting that radar independently now of the human. It is similar to suicide technology currently used in Ukraine. For years, there has been speculation about whether it could be combined with facial recognition software. Where we want to go, uh as we progress, is for autonomous perceptions, machines
that are able to reason. If you have a human that's being next to a building and then the human disappears, we would like the machine to be able to describe that the human went into the building and that the number of people in that building is now greater by one. Eventually, we would like for machines to actually be able to learn without our supervision and perhaps even figure out what they should learn and navigate and learn independently. In the near future, machines are expected to become full-fledged team members and also cooperate with each other. Research is being conducted on drone swarms in particular. They could significantly increase an army's offensive power.
The progress in the area of cooperation is enormous. What is much more difficult to teach machines are the ethics that go along with it. AJung Moon is a roboticist at McGill University in Montreal. She conducts research on human-machine interactions and runs the RAISE lab, which focuses on robot ethics. Lethal autonomous weapon systems can be used to physically harm human beings in undignifying ways uh in mass at scale. When we are looking at the act of killing someone or taking somebody's life uh some people might argue that dying is
a same thing regardless of the manner in which you die. Some people might argue, "No, that's not true. There is an undignified way versus not. With ethical issues often taking a backseat, Arjun Moon has been pushing for stronger regulation of autonomous weapons for years. It is a race against time. Lethal autonomous weapons, I believe, are real threats. Um I believe they are threats of today's warfare. I believe that it will become even worse than we have the imaginations to think about today. That is also why I think regulatory schemes and leadership is really needed because we need a global movement, bottom-up agreement that such a future
is not the kind that we want to envision. Alberto Pellicione is co-founder of Reactor, a cybersecurity company recently acquired by IBM. Reactor uses artificial intelligence to stop hacker attacks. Pellicione believes AI can make the world safer, but he knows software can be used differently. It's a realization he to during his time with one of Europe's most notorious hacking firms. More and more reports surfaced linking Hacking Team software to operations against activists and journalists.
Just as it did 10 years ago in the cyber weapons market, a wave of change is rolling through the robotics market today. That change is driven by emerging tech and a new global security situation. The Pentagon has drawn numerous defense contractors into its orbit. What does a military in the year 2030 look like? AI should be piloting and maneuvering everything, right? There should be an AI pilot maneuvering aircraft, there should be an AI pilot maneuvering destroyers, cruisers, aircraft carriers, there should be an AI pilot maneuvering submarines. Shield AI is an American defense startup.
It was launched in 2015 with $100,000 in capital. One of the founders is Brandon Singh. Shield AI is building the world's best AI pilot. Now, what is an AI pilot? You can think of an AI pilot as an AI that commands and maneuvers an aircraft. Very similar to self-driving technology, but we're applying self-driving technology to the aviation market. The AI executes the mission completely on its own. It has a higher level task and it figures out how to execute that. A perfect example was our very first product, which was a quadcopter with our AI pilot, which we call Hivemind.
It would go inside buildings where you would, you know, there's no GPS inside buildings. You'd lose communications quickly depending on the size of the building. But Hivemind would fly the aircraft through all the doors, through different windows. When communications was available, it would tell the people what was inside before they went in. And it's doing this by fusing a lot of different sensors, processing that sensor information, and then reasoning about what it should do. A former Navy SEAL, Singh has tailored his product specifically for the military. Shield AI secured its first Pentagon contract just 1 year after launching.
Air Force projects followed and new offices in the United Arab Emirates. Now, Shield AI plans to integrate its AI pilot into full-scale fighter jets by the end of 2023. AI enables an aircraft to maneuver without the need for GPS, without the need for communications. And then it can do all this with superhuman ability. Fighter pilots can only withstand so many Gs. An unmanned system is not bound by some of the human limitations that we have.
Today, the company is estimated to be worth 2.3 billion dollars. The market potential is enormous. While countless AI programs are underway in the USA, Europe and Germany in particular have so far been hesitant. In the field of related technologies, however, German products are in high demand. Hensoldt sensor technology can be found in numerous weapon systems from tanks and helicopters to aircraft and border protection systems. It also builds massive radar systems for combat ships. The company is probably best known for its contributions to the Eurofighter and the IRIS-T SLM, an air defense system that was recently supplied to Ukraine.
Future fighter jets will increasingly be integrated into a network of other systems. The amount of data involved is mind-boggling. Connected weapon systems such as the FCAS would be virtually inconceivable without AI. That makes artificial intelligence indispensable for armaments manufacturers. But, is it also reliable? When we assume that robots can be the perfect rule followers, we also assume that the real world is as clean and easy for us to operate in as simulation environments. In reality, when it snows in Montreal, autonomous cars and using sensors there not a very good idea.
In the first semester of my mechatronics engineering training was really about putting LEGO robots together and learning what a robot is. Essentially, sensors, processing, and actuators. Uh gradually I learned that I can actually build very weird things and nobody will tell me anything about it. If I were to hook up a sensor and then get it to shoot a gun or stab somebody, there are no rules against building such a system. And I thought that was really strange. Because of her concerns, Art Chon Moon decided against entering the robotics industry after graduation.
Instead, she dedicated herself to fighting for the regulation of autonomous weapons. She works as an expert for the Canadian government and advises the United Nations on its regulatory process. Although academia and civil society organizations advocate for greater controls on autonomous weapons, it is often difficult to convince governments of the dangers. While smaller states, in particular, are arguing for stricter regulation of killer robots, world powers such as China, India, Russia, and the United States have other interests. It seems as if we're moving toward a new, ever more aggressive, arms race.
Today, the PRC is the pacing challenge for the Department of Defense. Only through the integration of AI and related technologies will we gain both an information advantage and operational advantage on rapidly accelerating future battlefields. By 2025, they will be better than us. And by 2030, they will dominate the industries of AI. China, uh from what I understand, uh has some approaches that America is not necessarily willing to implement, right? I think the fears of killer robots are probably a little bit ahead of our time. We probably don't need to worry about that quite yet. My concern is getting ready fast enough.
Uh I don't think we need to be the fastest. I think we need to be fast enough, and we need to be better. People worry so much about making mistakes that it takes a very long time for us to get things right sometimes. We're used to testing, you know, tanks or helmets or rifles and being able to test them under every condition that's of interest, that's no longer the case for AI. And so, technology that is obviously more risky, we have to become more willing to accept risk. That's another thing that keeps me up at night is that we won't be able to take advantage of this amazing transformative technology because we have these processes that were put in place for good reason, uh but perhaps it's time for a change.
Ukraine, March the 17th, 2023. The 387th day in the war. The result of Iranian-made Russian suicide drones. For months, there have been attempts to wear down both Ukraine's infrastructure and its people. Further east, trenches crisscross the countryside. They are frighteningly reminiscent of World War I. The Ukrainians defend their country against Russian superiority by any means necessary. Commercially available drones and robots are becoming a crucial part of their fight. If you're under attack and you want to protect your country, your family, your community, who wouldn't look for an edge to protect? But at the end of the day, we also
forget that the technology can be an equalizer. One of the unique things about the topic of lethal autonomous weapons is that it depends so much on software. So, unlike nuclear war, where we can trace specific kind of physical substances that are required to develop these systems. Lethal autonomous systems, they run on software that can be downloaded. You just need some internet connection or some sort of connection of any sort, and you should be able to actually deploy the software of your systems anywhere in the world. Well, how are you going to do export control of that? The power of software and the transferability or modifiability of these software becomes a big game changer. And it also poses the issue that it
doesn't have to be states entering into warfare. It can be any kind of terrorist groups getting their hands on hardware with software on board, and they'll be able to do much harm. Think about autonomous cars. Anybody can hack into a system and use it as a bomb-carrying weapon in the middle of a dense city. Who's going to stop that? The new weapons revolution poses many new dangers. Intelligent software makes technology more accessible, but also equally vulnerable. And it's not just machines that are vulnerable to hackers.
It's also the human brain. After leaving Hacking Team, Alberto Pellicione decided to develop software purely for defense purposes. But while 10 years ago data theft and surveillance were the main concerns, AI is currently revolutionizing an adjacent field, cognitive warfare, the hacking of our minds. At its core, the concept is not new. Whether it's propaganda, advertising, or psyops, it's all about influencing the mind, in war as in peace. What is new is the efficiency of modern algorithms and how they are changing our perception. The machine doesn't care whether we are customers or enemies.
Brain is and will be the 21st century battle scape in many ways. End of story. Targeting, deep fakes, AI generated disinformation. Humans are data food and a hungry algorithm can never have enough. In the 21st century, warfare is about winning the information the decision space either before or during a conflict. This is the deciding factor. The more the algorithm knows about us and feeds on us, the more it can target us. It is therefore no coincidence that populous countries with weak privacy policies are living laboratories for developers.
Complex models of human behavior also open up new ways of waging war. The Chinese army is already talking about how AI could be used to take Taiwan without the use of force. Lawmakers from Taiwan's ruling party accused China of conducting so-called cognitive warfare through its social media to influence young people here. These regimes actively conduct influence operations to erode our confidence in democratic institutions and freedom. Russia also knows about the value of the cognitive domain.
Doubts are being deliberately stoked to undermine popular support for Western arms deliveries to Ukraine. Of course, the West counters with its own information campaigns. But in countries of the global south, they are met with historic skepticism. We are I suppose it bestandteil eines demokratischen westlichen Wertesystems neigen dazu unsere Kriege als sauber zu betrachten. Frage ist, ob die davon betroffen sind, das auch so sehen, ja? Das Ergebnis erkennen wir dann immer wieder. Ich habe ja insgesamt einige Einsätze schon absolviert, die war also paar mal am Balkan im Einsatz.
Da war in Afghanistan, in Mali, im Tschad, Zentralafrika, dabei auch kurz einmal im Irak. Und was hat sich da durchzieht, ist das egal, welche Rolle du hast, es vor allem auch um die Kommunikation geht, die Art und Weise, wie du kommunizierst und auch die gegenseitige Wahrnehmung und Wertschätzung. Das Afghanistan war z.B. so ein Moment, wo ich erkannt habe, dass quasi der Schlüssel nur im Umstand der ansässigen Bevölkerung zu finden ist, wenn man jemals in der Lage sein möchte, dieses Land in einer neuen Art und Weise zu befrieden. Die war damals eingesetzt im Bereich Psyops. Da geht es also darum zu versuchen zu verstehen, wie denkt der andere und wo sind quasi Ansatzpunkte, um ihn davon zu überzeugen, field camps and drones are
becoming the means of communication for the international alliance. Humans are increasingly replaced by machines as the face of Western values. I'll lead off with Hellfire or over 30 mm left and right Okay. Hellfire, you're clear to fire? Und damit natürlich kommt es zu einem Bruch zwischen der Bevölkerung und dem Military operations seem to prove that robotic systems are ultimately unsuitable for resolving conflicts in a sustainable manner. Nevertheless, that dehumanized security paradigm was eagerly brought back home.
A hostage situation needs to be resolved. But responders don't know what's happening inside. They decide to use experimental equipment. It's a decision that sends shock waves. That thing is creepy. I think this example of Boston Dynamics doing a partnership with the NYPD is a perfect example of why the power dynamics matter. At the time, there is all sorts of discussions going on about the relationship of the NYPD with the citizens of New York City. Deploying robots into that context means that context is attached. If you put a robotic dog on the streets of a wealthy neighborhood to police that neighborhood explicitly,
the backlash the police department face would be immense. So, they chose a neighborhood with people who don't have that much power in society. They framed it as this will help us solve crime in these communities while ignoring all this all the other like underlying actual causes of crime. Crime has never been caused by the lack of robot dogs. It was just another example of a new technology being created and it immediately being used to oppress people. If you are wanting to truly put your robots to use in good and in really build like a positive profile, that may
have not been the best time and place to do so. And I think this is another reason why it's good to put these discussions out there because I think it's important for a given police department to have these conversations if it's appropriate to deploy a robot and in what context are they appropriate. A wave of outrage ended the NYPD's collaboration with Boston Dynamics. For now. But the technology is already being used in other police departments. In Germany, too. Along with Boston Dynamics, Ghost Robotics is one of the best-known manufacturers of legged robots in the US. Instead of doing dance videos, their robot dog went viral in a 2021 image with a precision gun mounted on its back.
What started as a university project evolved into something much bigger when the platform was commercialized. The robot was becoming more robust, stronger, and weatherproof. And it had to have the longest possible battery life. It was an approach that quickly attracted a new customer base. The military invested in early versions, like it did with many other companies. They just happen to be able to pay for it. The main use case for the Air Force was something that none of us thought would be useful, which is patrolling part of their base. So, these are large installations, hundreds of acres.
It's got rocks and bushes and trees. And, believe it or not, people do wander on. In a similar use, the robot dogs are also being tested for border patrols. But the technology is attracting interest in other areas as well. Many of the other government agencies are looking at the robot. You know, the Marines, army, as well as some of the three-letter agencies. But the robot is categorized as EAR99. And what that means is we can technically ship the robot anywhere. Although Ghost Robotics doesn't develop weapons itself, it does not rule out arming their platform. The US government that supported our, you know, our existence, they have a right to request what they want. We don't have the capability of making
something like that decision, which is where we look more to, you know, policy makers and policies that dictate and govern how these technologies can and should be used. Ukraine, June 2023. Over 460 days at war. A counteroffensive is underway. But it is stuck in the Russian minefields. There, Ukrainian tanks are easy targets for artillery, helicopters, and suicide drones. Despite high-tech Western weapons, everything seems to be the same as ever. People die. People kill. Machines are assisting them.
Despite all the promises, AI has not changed the essence of war. Intelligent systems that can merge information and sensor data more efficiently are already used in Ukraine. So far, however, it has only sped up the killing. One example is software from the US company Palantir. Our role in Ukraine has been widely discussed. Won't say that much about it, but the old way of targeting where you don't use you where algorithms aren't used is clearly a failure. If you go into battle with old school technology, even if you're spending 65 billion a year and you're highly accomplished war fighters like Russia, and you have an adversary that knows how to install and implement digitalized targeting and AI, you
obviously are at a massive disadvantage. In the right hands, targeting software is a powerful tool. But the algorithms needed for it are highly complex and difficult to train. That's because they have to follow countless rules, military, ethical, and international law. In order to prevent the machine from making catastrophic mistakes, a human must always take the ultimate responsibility for all life and death decisions. We have one 2S19 in the open. In the design of these systems, we need to ensure that humans can meaningfully understand what is happening.
Does the human simply become a cog in an autonomous killing machine? I'm not so worried about the Terminator scenario. You know, I'm worried about things like flash wars, for example. The concept of a flash war was originally taken from the from a flash crash, which is when you have a stock market which has a lot of algorithms participating in trading and something happens, some unforeseen event, and the algorithms crash the stock market. The idea of flash wars is quasi does Flash wars, the total loss of control, is a horror scenario that must be prevented.
The major powers certainly see the risks of this technological development, but escalating saber-rattling could elevate the danger to the next level, the nuclear level. Sometimes we're in a situation where everybody is partially responsible for everything. When it comes to these weapon systems, I think we're headed in that direction. There has to be the sense of stewardship that whatever we design, we design responsibly.
