https://www.youtube.com/live/rRRNt0o00Rc?t=26560

Can Democracy Survive AI?
Bled Strategic Forum — panel discussion Moderator: Nik Gowing

Panel

Mihăiţă Blaj — Undersecretary, Secretariat of State, the Holy See
Oleksandr V. Danylyuk — Associate Fellow, Royal United Services Institute, United Kingdom
Michael Harasimowicz — Director of Artificial Intelligence, Lockheed Martin Corporation
Matt Jacobs — Senior Advisor, Centre for European Perspective, Slovenia
Aleks Jakulin — Philosopher-builder; researcher, professor and founder of data.Flowers
Michaela Küchler — Secretary General, International Holocaust Remembrance Alliance
Renata Salecl — Philosopher and Sociologist; Senior Researcher, Institute of Criminology, Faculty of Law, Ljubljana
Also speaking from the floor: Marko Grobelnik (Jožef Stefan Institute), a second questioner Mark Pleško, and Sotiris (representing a cluster from Canada).

Speaker labels in the source transcript drifted several times mid-session — the same person was assigned different numbers in different stretches. Attribution below has been reconstructed from context. Aleks Jakulin's remarks have been edited for clarity and concision while preserving their meaning; other speakers follow the supplied transcript.

Nik Gowing: Right, good afternoon, everybody. We're 57 minutes to run, so we're losing about 3 minutes at the moment, which is quite critical because you can see I have a panel of 7 and you can do the arithmetic. But this is about "Can democracy survive AI?" Let me just give you some opening remarks. We are talking about, here, potentially a tsunami of overwhelming vulnerabilities. Remember what the Prime Minister said yesterday? He said, "We are in a world of AI." The Foreign Minister said, "There are many alternatives to liberal democracy." We must share dialogue with those who do not share our values. So it's easy to talk about democracy, but we need to all define what we mean by democracy. The framing underpinning our discussion, I think, is the debate about protection versus regulation. Big Tech is asking preemptively for regulation, but out of scope of current EU pillars, as what came through yesterday in some of the exchanges. But let's all underline what Bill Gates published five days ago: the turbulent AI era is here. The most turbulent in human history, is how he expressed it. The choices we make now are critical. We need a plan to ensure the good outweighs the bad. Now, before we get going, and I'm going to ask each of the panelists to introduce themselves rather than me doing it, I want to just check with you: this is totally unscientific, but I'd like to ask you—let me shape the reminder with the argument—with two questions. Do you agree that democracy is threatened by AI? Those of you who think, yes, it is threatened, hands up, please. Those of you who think it's not threatened by AI, panelists, you have an idea that it's threatened by AI. Secondly, so can democracy survive AI? Yes or no? Yes, please. Democracy can survive AI, can't survive AI. So that should inform—it's totally unscientific, I know—that should inform everything that we're talking about here. But I think it's noteworthy that The Economist last week had a very stark image here of how AI is breaking the state. It talks vividly about how AI is leading to populism and leading to people pressuring the state in ways which they've never done before. We can talk about that shortly. So that is the outline. Underpinning our discussion is protection, and also the kind of things which people are thinking about at the moment, the kind of things which are worrying people, the kind of things that are reassuring people. Are people understanding even what AI is? That, it seems to me, is critical in all of this. So those are the framings that I want to put on the agenda for the start. Let me tell you, nothing has been rigged by AI for this discussion. This is all human beings talking to you. We've talked to each other by email beforehand, but nothing has been fixed by ChatGPT or Grok or Claude or anything like that. So be reassured, you're humans talking to humans. We can laugh about it, but you never know these days. Right, each of you, I'd like to explain what you do. Michaela.

Michaela Küchler: Thank you very much.

Nik Gowing: You have a minute each at the moment.

Michaela Küchler: Thank you very much. My name is Michaela Küchler. I'm the Secretary General of the International Holocaust Remembrance Alliance. That is an intergovernmental organization of 35 member countries, 8 countries wanting to join, and 10 international partners. So we're a pretty big organization. And you might wonder why.

Nik Gowing: And what's your experience with AI?

Michaela Küchler: Well, you might wonder why the International Holocaust Remembrance Alliance is dealing with the question if AI can threaten democracy. But AI is affecting the work of the IHRA, the International Holocaust Remembrance Alliance, in quite many ways. It helps Holocaust remembrance because it helps researchers to work through large amounts of archives and to find and make it easier accessible information for the user, for historians. It helps educators create content that is trustworthy, based on facts, and that speaks to young people, like in animated films or in comics. But on the other hand, and that is the worrying part of it, AI also makes it possible to create very convincing fake images, fake news, fake messages, fake testimony.

Nik Gowing: Can you identify that?

Michaela Küchler: Yes. For instance, there was a TikTok video last year, documented by the BBC, by the way, of a girl standing in front of the gate in Auschwitz. The picture was completely a fake image. The story was completely fake. And it passed through as if it was—it looked as if it was real. And the guy who made it was sitting in Pakistan. When asked why he's doing pictures on the Holocaust, he said, "What is the Holocaust?" The only thing that he knew was it makes—he can make money with it.

Nik Gowing: Let's pause for the moment. Let's go to Alexander Danyliuk. Welcome, Alexander. You're from the Royal United Services Institute. Tell us what you do and what you're working on. But you also talked in your pre-discussion about the natural stupidity that's being created by AI.

Oleksandr Danylyuk: No, I didn't say that, actually, it was created by AI. It's just, like, what it is, right? So it's like—I mean, like, artificial intelligence is definitely, you know, a source of power. And because of it, we will actually see a dramatic, you know, shift in how things work globally. In fact, I was a part of the team working on this report, a classified one, four years ago, about the organization of the artificial intelligence by Russia. And I think that in this regard, our natural stupidity—or not our, but in general, the fact that people are getting less smart than they should be—it's a challenge, especially in the world in which you have these badwill actors like Russia, definitely is, who know how to use AI for different military and non-military ways to make damage to the rest of the world. And in this regard, I think that it's reasonable to say that we have actually three main areas in which AI is weaponized so far. So it's autonomous systems. And, like, a few days ago, there was another attack of this Molniya drone, you know, which used AI to kill civilians. So this is our reality. Second one, it's cyber operations, mostly cyber espionage and disruption. And the third is cognitive warfare. And in fact, Russians were quite good in conducting this warfare without any artificial intelligence. And now, for all three domains, they can be just much more efficient than they were.

Nik Gowing: Let's pause there for the moment. Let's go to Aleks Jakulin. Aleks.

Aleks Jakulin: I would describe myself as a philosopher-builder. I have worked as a researcher and professor, but I also build companies and AI systems. I think of AI as fire: you cannot slow it down.

Nik Gowing: Why fire? Why fire? I was interested in.

Aleks Jakulin: You cannot pretend it does not exist. You must master it, but combustion itself cannot be controlled; only the space around it can. We regulate how buildings are constructed, how fuel is stored, and how fire departments operate. We do not regulate fire itself.

Nik Gowing: Let's pick that up when we have a longer round. Matt Jacobs.

Matt Jacobs: Yeah, thank you, Nixo. My name is Matt Jacobs. I'm the Senior Advisor at the Centre for European Perspective here in Slovenia, NAPOCOM. And I work on, alongside my colleagues, on projects that help to build the capacities of government to adjust to technological changes in society so that they can better serve their citizens through using those emerging technologies in a way that benefits pro-social development and democratic resilience.

Nik Gowing: And you came here from?

Matt Jacobs: I came here from the Department of State, where I was the Senior Digital Coordinator for 15 years for the Bureau of European and Eurasian Affairs.

Nik Gowing: And how would you summarize the way state is acting on AI?

Matt Jacobs: I shouldn't really be speaking on behalf of the State Department on that issue.

Nik Gowing: But you give an impression.

Matt Jacobs: I would say that governments are looking at how to use these tools to increase productivity at the moment, but are not looking at how these tools will impact how citizens engage with information and what that will ultimately do to society when people learn to distrust, trust, or generate information in different ways.

Nik Gowing: Are we moving in optimistic directions, positive directions?

Matt Jacobs: I'm entirely optimistic about this. Absolutely.

Nik Gowing: All right, remember that.

Matt Jacobs: Yeah, yeah, yeah. When can democracy survive AI? Yeah. Yeah, yeah. It'll be fine.

Nik Gowing: Mihăiţă Blaj, welcome.

Mihăiţă Blaj: Thank you. Yes, I'm the Under Secretary for Relations with States in the Holy See. That is sort of Deputy Minister of Foreign Affairs. We are running the diplomatic activity of the Holy See under the guidance and together with the Holy Father. Of course, I suppose you wouldn't expect that the Vatican is extremely rapidly using artificial intelligence, not because we tend to be rather conservative, but especially because we are too in love with human beings so that we fell in love immediately with something that is artificial. But anyway, we are dealing every day with that issue, with this question, especially as a matter of reflection, of engaging with state actors, with actors from high tech, because it's part of our world. And the fact that the Pope Leo decided to write his first encyclical on the human person in the time of artificial intelligence, it is also because we are convinced that this is not only an invention. This is something different. Since it is imitating our intelligence, the artificial intelligence is able to penetrate everything we are doing.

Nik Gowing: Are you challenged by fakes?

Mihăiţă Blaj: Well, of course, we are all challenged by fakes. But I think it's not the main threat, because on that level, we have some instruments already, because we have two decades of use of social media. So we have some experience on that. We have some instruments. I think there are other problems that are directly connected to that artificial intelligence that need to be addressed.

Nik Gowing: Right. We'll hear about that later. Renata.

Renata Salecl: Yes. I'm Renata Salecl. I'm a philosopher and sociologist from the Institute of Criminology at the Faculty of Law here in Ljubljana. I also teach at Birkbeck College in London. And I have been working on ignorance, The Passion for Ignorance is my last book, and on rudeness, especially how rudeness is now kind of part of the public discourse, especially encouraged also by AI, by the social media. At our institute, we look at the legal, ethical issues in regard to AI, and especially at the psychological issue.

Nik Gowing: And your summary, the conclusion you have at the moment?

Renata Salecl: I'm pessimistic about the survival of democracy, especially the human rights. And I think that the psychological dimension is completely negated, and also the cultural dimension. So we are working on how AI will reshape arts culture at the moment. And here we see that legislation is only concerned with consumers and kind of, let's say, corporations at the moment.

Nik Gowing: Right. Michael.

Michael Harasimowicz: So my name is Michael Harasimowicz. I'm a Director of AI at Lockheed Martin. I started my journey in AI back in 1989. I know that's a long, long time ago.

Nik Gowing: Was it called AI then?

Michael Harasimowicz: It was, as a matter of fact. We used natural language processing to take a look at really difficult, beyond-human scale text analytics and create insights from that. I begged the Air Force I was in the Air Force, and I begged them to say, "Hey, leave me as an engineer in this assignment." We all filled out a dream sheet, and they said, "No, we're going to send you somewhere else." And I'm like, "AI is going to be big someday." I really wanted to be in this field. So I did a 25-year stint in the Air Force within intelligence and cybersecurity and flying operations and had a wonderful time. But as soon as I retired in 2015, I went to JP Morgan, America's largest bank, and helped stand up their AI system in 2015. So that's now 11 years ago. We put multiple systems into production to help protect a customer, to help understand cyber threats, to combat against insider threats. So very, you know, defensive kind of mindset level of activity. That AI was very effective. You know, we could create efficiencies and viable signal when there was a lot of noisy data. So AI was a very viable player, again, 11 years ago, before ChatGPT and Claude and everything. And now, within Lockheed Martin, we're taking that same mindset. And we've been using AI from decision trees and knowledge graphs, those type of things, for over a decade, and putting into systems to help understand the environment. With a lot of information coming in, there's always a lot of noise. How do you elevate the signal? How do we prepare our operators to have better understanding of what's happening in the world?

Nik Gowing: And do you believe that there is a full understanding of what the hell AI is? I mean, you're someone who I couldn't, if I can put it politely, a geek. You've been at it for a long, long time. There are many who are struggling to try and work out how this works, how it affects them, how they can make it operate, even ChatGPT.

Michael Harasimowicz: The way I look at it is it is a tool that you have to assert your dominion over. You have to understand, you have to have a certain level of subject matter expertise. So there's a level of domain expertise. Then to make AI functional, you have to instruct it, correct it, and make it better every day. And again, the human has to be the one that's in charge. A lot of conversations I'm having with folks out there is, "Wow, AI is more accurate, faster. It's more charismatic. It's more creative. It's more empathetic." All those human functions all of a sudden now become, "Well, I'm just going to let AI do that difficult or that messy thing for me, because I'll just do something else that I enjoy." I think we have to be the best human possible. And I love that Pope Leo put out his first encyclical, Magnifica Humanitas, that says, "Hey, the human has to be first in this. We have to preserve our faces and our voices."

Nik Gowing: Let's pause there for the moment. Let's get more detail from each of you. Now you've introduced yourself. Everyone's aware of the kind of things you're doing. We could have tens of thousands of people up on this platform. Seven is enough for an hour, particularly if we want to get a plane later. But let me ask you to give more detail, Mihăiţă, particularly about trying to work out what is false and what is genuine. And that's something you emphasized in the pre-meet that we had.

Michaela Küchler: Yeah. Well, we think, I think, the problem with AI is double-faced. So on the one side, you start believing fake images, fake stories, fake narratives. And on the other side, you start to doubt genuine evidence. And that is a real problem, because that undermines trust in everything that is being written or produced as pictures, films, or whatever. And when trust is being undermined, then we have a problem also for democracy, because democracy relies on trust. It relies on citizens who can make judgments, who can make decisions on the basis of facts, and of facts they can trust. And if that trust is being undermined by AI from two sides, by creating fake images and by doubting genuine images, then democracy is in really heavy waters. And we will have to find ways to and I fully agree with what you have said. Human must come first, and man, humankind must dominate AI.

Nik Gowing: How do you persuade people to go to reliable sources, sources which are clearly good information, as opposed to the kind of fakery that you just talked about, about that image from Auschwitz?

Michaela Küchler: Well, we have a huge network of well, we have 35 member countries. In these 35 member countries, we have a huge network of educators, researchers, and so on, who help us multiply the message. And you report back to us also, who give us feedback on what is happening out there and finding ways to accommodate all these problems. So it's our network that helps us to promote the understanding of genuine evidence and of genuine sources. For a very long period, I would say for the last 30 or 40 years, we had the most reliable source we can have, and that is the people who survived the Holocaust. This source will soon be not be here anymore, because Holocaust survivors are passing away. They are very old. And so our main sources now are sites where the Holocaust happened, concentration camps, other sites, and archives. And we work heavily on making these sources accessible in a way that people can trust this information.

Nik Gowing: Right. Let's pause for the moment. Alexander, let me come to you and pick up the point you were making earlier, particularly about stupidity. And I'm just emphasizing that because you wanted to emphasize it in the early remarks. But how much are you taking this forward now, getting an appreciation, a comprehension of just how profound these challenges are and the speed at which it's happening, given what Bill Gates is saying?

Oleksandr Danylyuk: Yeah, I think that, in fact, this phenomena is the basis, you know, for all of the problems. And it's not like we can heal it, right? So that's why it's natural, right? And it's also quite clear that artificial intelligence, as well as nuclear weapon or any other technologies, they are not dangerous by themselves, right? So it's just because you have badwill actors and they can use these technologies to do harm, you know, to the rest of the world better, faster, more efficiently. And this is what we need to take into consideration here, because, OK, so Russians were quite good with fakes, you know, and influencing societies and governments without any artificial intelligence. Rumors were used, you know, as a weapon of political warfare, you know, thousands of years, right? So the problem is that we, if we believe that we are good guys, right, and we have actually this moral superiority, we should be able to influence behavior of these badwill actors to prevent them from doing damage with artificial intelligence or without. And of course, the problem is that we don't have this, you know, toolkit which would give us this power. And artificial intelligence could be used both for, you know, good and bad things, like any technologies. And I think that I agree that pretend that actually it doesn't exist, it's too late. It exists. And what we need to do is actually to be better than our enemies and to be stronger than them and be able to shape their behavior according to the norms of international law and our values. And yeah, it was quite questionable if actually peaceful coexistence existed, you know, after the end of the Second World War. Today, the world is too global with social media, other things. You're absolutely right. So like, you can, with or without artificial intelligence, you know, you can penetrate societies, you can influence them. Look what's happening in Ukrainian-Polish relations right now. So we have actually attacks of some football hooligans, et cetera, on Ukrainian-speaking people in Poland right now. And partially, it's influenced by these videos created with artificial intelligence, but they could be created without.

Nik Gowing: Now, Alexander, what you highlighted is the revolt of the masses. What do you mean by that? Explain why you're saying that.

Oleksandr Danylyuk: I think that, you know, so we had these historical periods before, you know, several times. Usually, it was a result of each stage of information development of humanity. And unfortunately, great things like book printing, for instance, they were used for quite questionable, you know, outcomes. So that's why, you know, radio wasn't so bad thing, right? But as a result, Hitler came to power. So we can give a lot of examples how actually badwill actors, they are smarter and faster in seeing, you know, all of the advantages of new technologies and being able to use them immediately. And this mass, you know, culture, which is actually each stage getting more and more mass, it's a problem. And we have right now people who should be, you know, probably the most educated in human history, and they are not. So we have this situation that actually, because there is no such a thing as traditional media as the main source of information, people can actually watch and read whatever they want. And like, in fact, they want to watch and read different stupid things. So that's why it's another challenge for.

Nik Gowing: Who judges them to be stupid? Is it, I mean, you use that word. I mean, we know what you mean. But for some people, it's not stupid.

Oleksandr Danylyuk: Again, so it's like cognitive bias, you know. It's like nobody is totally immune to this, right? So like, we are all stupid in some ways. So that's why it's more about how governments, how societies, then they can provide, you know, proper education, not just about fake news, but in general, you know, because we have actually these people who believe in completely ridiculous ideas nowadays, mostly because they technically have not got a proper education, you know. And we can see it in Ukraine, especially because we had, you know, several years of distant education because of the pandemic and now because of the war. And of course, it's very difficult, you know, to, I wouldn't say indoctrinate, because it's not a thing we want to do. It's not the proper way to actually give some critical, you know, mindsets to our people and also to give them some basic information about how the world works, because quite often they actually have actually similar views as people had, you know, in the dark ages. So we, yeah, we are on the edge, I believe, of these digital dark ages right now, because.

Nik Gowing: Age of digital dark ages. All right. Well, let's go to Alex, and particularly to talk about governance and the AI governance being what you call a great major risk.

Aleks Jakulin: A week after OpenAI released ChatGPT, I was with some of its engineers testing it and discovering its unexpected behavior. We were exploring something that had emerged from humanity's collective output rather than a system anyone had explicitly designed. No one fully understands how it works, so we should treat its output with the same care as the food we consume. Education alone cannot keep pace; accountability for the information we publish is essential.

Nik Gowing: You use the word governance.

Aleks Jakulin: We should be careful about who governs these systems. Some people seeking that authority want to imprint their own ideology on AI, which could waste our opportunity to govern it well.

Nik Gowing: Wasting the opportunity. Why?

Aleks Jakulin: Governance requires understanding the technology. To govern fire, you need to understand combustion and fuel; talking abstractly about warmth is not enough. Many people trying to regulate AI do not understand how it works and treat it as a small artificial human. It is fundamentally different. An old AI quip says that asking whether a machine can think is like asking whether a submarine can swim. AI computes rather than thinks. Effective regulation should focus on the humans who build and use it, not on the AI itself.

Nik Gowing: Just before I go to Matt, and he's nodding at the moment, I have to ask you, how do you hang out with ChatGPT engineers? I'm intrigued as to how you do it.

Aleks Jakulin: They invited me to stay at their house while I was in San Francisco.

Nik Gowing: So it's physical as opposed to online.

Aleks Jakulin: Yes.

Nik Gowing: OK. I just want to check how you hang out.

Aleks Jakulin: They are very human.

Nik Gowing: That's what we want to hear. Do you want to describe any to us?

Aleks Jakulin: They are the canaries in the coal mine, which is why I value friendships with the people building this technology.

Nik Gowing: OK. Matt, let me come to you, because you actually are very positive, as you said earlier, but actually that everyone adapts eventually to this transformation, often far faster than anyone ever imagines.

Matt Jacobs: Absolutely. So, I mean, the thing is, is that we're talking about this technology as if it's about to turn around the corner. But the thing is, is if we look at it, I'm just going to quiz all of you. Who of you have not, who amongst you have not used ChatGPT or Anthropic or Claude? Raise your hand if you, any Luddites here?

Nik Gowing: No, don't be demeaning. Just don't call them a Luddite for not having done ChatGPT.

Matt Jacobs: So Luddite is a proper definition. I will tell anyone about that afterwards. But the thing is, is look, no one was.

Nik Gowing: Are you impressed? No one put up their hand.

Matt Jacobs: No one put up their hand. No, I'm not impressed at all, because this is normal. These technologies, and we can look at it from, I mean, remember when Socrates thought the written word was going to destroy the human mind because we wouldn't be able to memorize anything? And Marseille thought that the printing press was going to destroy our ability to actually understand our local environment because of the spread of the word. The thing is, is that humans use these technologies as they come out. And as we see in this room, every single one of these people, I'm estimating about 200 people in this room, have used AI. And it's going to impact their daily life, whether it is something such as making a grocery list or whether it is preparing for a speech on stage or whether it is writing an email to a coworker. We are starting to figure out how to rely on these tools subtly. And these kind of small steps, these thousands of little attempts at using this technology is what starts to really slowly redefine society until it gets to the point where, you know, Weber said that it becomes indispensable, inseparable from society. Our society itself adjusts to the millions of little movements that we take every day. And if AI is one of those millions of little movements, then we are going to adjust to that. And that doesn't mean that there's not going to be consequences. The radio did propel Hitler. The printing press did arguably, you know, contribute to the Thirty Years' War. Can't really say if Socrates was wrong. We're still out on that one. He seems to be pretty respected. But there will be these changes within society, and they will precipitate friction. They will precipitate and, you know, challenges that we have to overcome. But will we survive it? Of course.

Nik Gowing: But you made a point that actually there's an assumption that needs to be a kind of what you call moral panic.

Matt Jacobs: Yes.

Nik Gowing: Why is that not the right framing to be thinking or to be far more optimistic, in other words?

Matt Jacobs: So the reason I think that it is a moral panic is because there hasn't been, as a technology that hasn't precipitated this fear of the status quo changing as a result of this technology. Each of these technologies that I've addressed, whether it's the printing press, the written word, AI, radio, essentially expanded human capacity. It expanded our limitations as individuals to communicate more broadly, to share more information with each other, whether that is writing something down, whether that is plugging something into OpenAI and paying for premium so you get an invitation to crash at their house. We use these tools in a way where we feel that this might shift our society. If we can use AI to change how we get information, it might change expertise. It might have all these different implications. But we have this panic that suddenly this is going to be the thing that unravels everything we've created. And we've been here before. We had this same conversation with social media. We were still having this conversation about social media down the hall an hour ago in Sunset. We, as humans, as society, we panic about these things.

Nik Gowing: But why do we, why do you say we're better prepared to adapt to this new technology than previous technologies?

Matt Jacobs: Oh, absolutely. So one of the things.

Nik Gowing: Why?

Matt Jacobs: Because it is happening at a faster and faster pace. We've already had to adjust to different technologies. Within this one generation, we've had to go adjust to the 24-hour news cycle. We've had to adjust to the democratization of information production through the digital, you know, information age. We have lived through these changes, especially people who are in their 20s and their 30s. They're going to be so much better suitable or suited to adapt to any sort of change that comes from AI and precipitating changes that come from technologies that follow AI in order to adjust for it.

Nik Gowing: Thanks. The moment, let's go to the Holy See. Picking up what you were saying earlier, Bludge, about where this is going, about the need for truth and accuracy. What have you been discovering, particularly under the new pope?

Mihăiţă Blaj: Well, generally, the tools, the instruments, the things are morally neutral. That means this bottle, if I use it to drink water, is positive. If I use it to throw it into your head, it's negative. So it's the use defining the morality of things and instruments. It should be the case for the artificial intelligence as well, as well. But we have a problem, the timing. The artificial intelligence is entering our world when our world, excuse my French, is the jungle. So that means that it is imbued with competitiveness, with greed, with deterrence, with the will to control, et cetera. So there is a lot of positivity, and I think it's not necessary to talk too much about that because it's sufficient to let it developing. So this is part already, and it will be, it's amazing. It's amazing what it's able to do, and we agree with that. But we should have some concerns regarding what is not so good. So I think if we stick to the link between the artificial intelligence in democracy, we have, I would say, three major challenges. The major challenge, the first challenge is on the domestic use of the artificial intelligence. It is not only the issue of propaganda and fake news, which is a big part of that. It's the fact that the use of the artificial intelligence is challenging our being humans in our relationship with the truth. The fact that with one click I have access to what I need in that moment has a huge impact on my ability to do research, to engage with others, to cross the sources, to reflect critically, et cetera. This is not having a huge impact on us. We are already structured persons. But I should, we should be concerned regarding the next generations that are going to use more and more this tool because of that. Another level is the level of the job market. It's destabilizing potentially our societies, so also threatening our democracy, not only because it's new, that's normal, but because it's fast. Being too fast, I doubt that our governments and societies are able to adapt as quickly as artificial intelligence is ready already to replace the jobs in our societies. And firstly, it's the link with war industry. And that is where we are really worried about that. And it is where we need more and more regulation.

Nik Gowing: Now, you've signed up to talking about can democracy survive. Now we can have an interesting discussion about the state of democracy within the Holy See and the church. And I'm not being sarcastic here. Every community has its own rules and regulations. Do you feel a community under pressure and under challenge and under tension because of these new realities?

Mihăiţă Blaj: Well, there is, there is a pressure, of course. But I would speak rather on governance, as other colleagues spoke about.

Nik Gowing: But are you changing how you govern within the Holy See?

Mihăiţă Blaj: Well, I think when we speak about artificial intelligence, it's not sufficient the national governance. It has to be internationally. You can close your door. You can control your back door. You cannot close all the windows in our world. So that's why I was talking about the timing. We are in an era where the international collaboration is very low. So if we are not able to collaborate internationally in order to have a good governance about that, this is a major challenge. We proposed as a Holy See in 2021 to institute an international organization for the artificial intelligence. And we proposed to negotiate a treaty for the governance of the artificial intelligence, especially linked with the use with weapons.

Nik Gowing: All right. Now, Renata and Michael are going to speak, and then I want to open it up as well. We'll have about 15 minutes. We're making good time, even with seven speakers. So you've done incredibly well. Thank you very much indeed. So do, if you've got ideas you want to put into the mix, please store them up because the microphone will come to you, Renata, particularly the manipulation of truth.

Renata Salecl: Yes. I think, you know, this discussion about stupidity is very important because in some ways stupidity is encouraged. I'm not saying by the developers of AI, but many who are using it, actually the fact that people are becoming less critical might be encouraged politically. For some, you know, it's much easier to manipulate people when they are thinking less. Recently, I was traveling as a tourist. By chance, I met an entrepreneur from the Gulf States who runs five companies, and he says how wonderful it is now he can travel as much as he wants because he controls every worker via AI. Every hour, all his workers have to.

Nik Gowing: Controls.

Renata Salecl: Controls. Controls. Every hour, every worker has to send to an AI organized system a picture of what they are doing, a photo of what they are doing. And then AI analyzes whether they were lazy or not. And then, you know, he said how wonderful it is. There were one human, he says, in the loop. And then I said, you know, but with AI, it's a problem. People are starting to think less. And he said, I don't want workers to think. There were three teachers nearby there, and I said, but teachers have problems. They can't teach students anymore because of AI. And he said, I hate teachers. And then I said, you have three children. Do you hate teachers who teach your kids? No, I want my kids to learn, but I don't want my workers to learn more than it's necessary.

Nik Gowing: Were you shocked?

Renata Salecl: I was shocked.

Nik Gowing: Do you think that's general?

Renata Salecl: It was honesty that I didn't yet discover, you know, that people would openly say, I don't want people to think. And I think here this desire that people think less is in a hidden way embedded in the way AI works. AI is a mirror. Often it complements your idea. It says how wonderful you are. Do you want a bit more? There is no distance. Democracy requires distance also from another. We need to understand a different view that another has, not just massaging of our ego. So more and more people have a problem dealing with people who have different opinions than themselves. That has political implications. We are very divided societies. And the next problem is we are now in times where people cannot afford not to use AI. American students were analyzed. Do they like AI? Majority hated it, but they are using it all the time because they will fall behind others because they said everyone else is using them. What they said also is we need to learn a new how to think. And that's the problem. Whether we will be able to reflect. I'm sure that, you know, the children of your friends from ChatGPT are not allowed to use too early AI. I met some people, you know, from Google who school their kids in London privately because they don't want too much technology to come into their lives too early.

Nik Gowing: Alex, what's your answer to that?

Aleks Jakulin: History offers examples: Japan banned guns and was conquered by the United States; the Ottoman Empire banned printing and was eventually defeated. Banning technology is not viable. Countries that try risk banishing themselves, so we need to engage and learn how to work with it.

Nik Gowing: Matt, I mean, you've been so optimistic right at the other end of the loop here. What's your reaction to what you've just heard from Renata?

Matt Jacobs: I mean, I don't entirely disagree with that. I might be optimistic, but that doesn't mean I'm removed from the consequences.

Nik Gowing: Encourage not to think.

Matt Jacobs: Oh, no. I think that there is the hearing what that entrepreneur said, like, I don't know how I would have been able to stomach that situation. I think that's a terrible, a terrible sentiment. But I do not believe that is probably a sentiment shared by the majority of people who are working in this space or who even want to use this technology to enable people. I'm a professor at a university where I deal with students who are using AI as part of their, you know, assisting them with their work. And it does become, as I said, Weber said, an indispensable part of society. But I do not believe that my students are not thinking when they're using ChatGPT to help them write things. In fact, I encourage them to show me how they would edit what ChatGPT said and to enhance what's going on. So if they're going to be using these technologies, same thing is like my math teacher used to tell me I wouldn't have a calculator on me all the time, so I better learn to do arithmetic. I have a calculator with me all the time, but I still learn to do arithmetic. We can have both.

Nik Gowing: Michael, just pick up all these points before I go to the audience. We've got participants, delegates. And who would like to come in?

Michael Harasimowicz: I'd love to.

Nik Gowing: Who would like to come in? Let me get a microphone to you, please. Jerry, where's the microphone? To here, please. Go ahead, Michael.

Michael Harasimowicz: So for the past 11 years, I've been building trust in our AI implementations, showing people how to use it, showing people where you can go wrong, giving that kind of guidelines and those governance and guardrails to help us use it the best. I transitioned about three years ago to changing to cautioning against overdependence and what we call automation bias. And that is deliberately to make sure that we reinforce the thinking, the critical thinking skills that the workforce needs to find efficiencies, to challenge the status quo. A human engaged in that is a productive human. In terms of the.

Nik Gowing: You talk about a human in a human machine team.

Michael Harasimowicz: Right.

Nik Gowing: Why do you put it like that? A human machine team.

Michael Harasimowicz: Because I consider it a combination. You want to have that level. Even though that CEO, you know, he's still a human in the loop. Somehow he's actually controlling everything. I think he's probably way over his skis on controlling everybody's life in his workforce. But you really have to develop the workforce by challenging them, giving them hard problems, and thinking that through. AI, again, is that tool that you have to be able to assert yourself over. And one thing that I wanted to stress to the audience is that there will be at-risk populations. And so to get to the main point about AI and democracy is that unfortunately, I think as the world has looked at information availability, one of my colleagues coined the term commoditizing truth. We've all been exposed to paywalls. Like, I want to know what Wall Street Journal is saying. And you're like, well, you can't unless you have a subscription. Same thing's happening with AI where you go, hey, I want the free version. Well, if you get the free version, is that the best version of truth, or do I have to pay for that? I find myself worried about that. And I have to reflect that you're hanging out with Google engineers. I'm actually hanging out with philosophers and sociologists more just to see what they think about this. And I've actually hired some to help us sort this out about what does it mean to really force humanity into this situation where we're going to be at a disadvantage. And we have to make sure that we're putting our best critical thinking, our, you know, cerebral cortex engaged because the human brain is really the ultimate equalizer. As big as these data centers are getting, and they're big, and they're not very attractive, they still consume so much power and, well, like 100,000 times more than, you know, our human brains.

Nik Gowing: All right, fine. Let's go here. Who else would like to come in? Anyone else? Microphone, you can get that one in a moment. Yeah, go ahead. Keep your remarks brief, can you?

Audience — Marko Grobelnik: Yeah. So my name is Marko Grobelnik. I'm head of local AI department. So going back to these statements, right? So AI is a mirror, right? What Renata was saying. Yes, this was true three years ago. This was the mirror. What we have today, really, AI, we can look through the mirror, right? I mean, just check what we have today, right? So I was asked a while ago, can we have one statement about AI? Yes, it was a mirror while it was simple LLM. Now with agentic AI, we actually looking through the mirror, right? So now, I mean, it was lots of negative stuff, what you said. Not everybody, but lots of.

Nik Gowing: Let's call it pragmatic.

Audience — Marko Grobelnik: No. Let's call it negative. You can call it pragmatic. It was not pragmatic. So on the good side, right? We should, the question is really what democracy can gain out of AI. So this is.

Nik Gowing: Do you believe it can gain from AI?

Audience — Marko Grobelnik: Should. I mean, otherwise we are dying. I mean, we can just close down the workshop.

Nik Gowing: All right. Let's leave it there for the moment. I'll leave them to pick it up.

Audience — Marko Grobelnik: So can you just say what it will contribute? Yeah.

Nik Gowing: What it will contribute. Pass the microphone to your neighbor. Go ahead.

Audience — Mark Pleško: Okay. Yeah. Hello. I'm Marko. I have a very simple or stupid question. I understood from the discussion that some of.

Nik Gowing: A simple question doesn't have to be stupid.

Audience — Mark Pleško: Okay. Fair enough. Let's judge afterwards. So it seems that some people are, you know, intelligent and they can judge, but there are many people that are apparently, you mentioned people are stupid and the CEO thinks these people are stupid. Then if we are so stupid as a race or as a species, don't we then deserve to be replaced by AI?

Nik Gowing: Michele, anyone else want to come in? Please.

Audience — Sotiris: Hi there. AI.

Nik Gowing: Introduce yourself.

Audience — Sotiris: Okay. I'm Sotiris. I'm representing a meta cluster from Canada. My engineers tell me this every day that AI has brought humanity in a crossroads similar to human cloning. And speaking of international collaboration, this is an issue that can be dealt in the same way. So my question to the military complex is this: would the military complex develop AI that is not going to be used against human soldiers during the time they sleep, when they drink water? And can this be accepted by the military complexes globally? Because, you know, for civil applications, we have all the time to develop. But in Ukraine, people are dying every day when hunters do not shoot animals while drinking water or while they're sleeping.

Nik Gowing: Let me put that to Alexander. As you're at the Royal United Services Institute, in other words, you're in a defense institute.

Oleksandr Danylyuk: Yeah. And security as well. Yeah. Yeah. I think that this is a very good point. And in this regard, again, I think that the main question should be if democracy can survive Russia, if democracy can survive, you know, some other countries having a completely different view on these things. And your question about if we can convince, you know, Lockheed Martin or any other, you know, Western defense industry companies to actually use these constraints, it's irrelevant because, again, so like the reality is that right now in Ukraine, we have actually AI-controlled drones killing human beings every day, including civilians. And this is the reality. And again, so like that's why my point was quite simple. So if we want to live in actually a better version of this world, a world which would be, you know, led by rules, by values we all share, we need to have, you know, our tools to influence behavior of international criminals like we do on our, you know, streets internally. If we don't have this toolkit, if we're not able to force, you know, badwill actors to behave, I think that the question just went.

Nik Gowing: Michael, do you want to come in on the Lockheed Martin issue? You can pass on it if you want.

Michael Harasimowicz: So we're designing systems that are well informed of laws of our conflict, Geneva Conventions, general principles, proportionality, the like. So there's that element that we and democracies live by a rule of law, and the way we're designing AI is operating within those same parameters. To describe AI as less discriminating, I think that would be false. I think the way we go back to history, and we've all had our share of stories, firebombing Japan was indiscriminate. The way AI is now discriminating is more effective to actually conduct mission operations. And ultimately, the human in the loop on that one is the US Code Title X sitting authority that makes those decisions.

Nik Gowing: You're saying there are weapon systems which can respect law, respect international law. Is that what you're saying?

Michael Harasimowicz: Built with understanding of that. Yes.

Nik Gowing: Wow. Michaela?

Michaela Küchler: Yeah. I wanted to come back to something that you, Alexander, said before, and that is about education and critical thinking and digital literacy. I think all that is designed for the individual. But can we really leave that burden to the individual? I think there is a big role for governments, international organizations, and international actors here to play, to regulate, to regulate AI in order that it will become something accurate, transparent. We might need some sort of labeling of AI if AI creates images or stories, accountability, and protection of genuine evidence. I think it cannot all be put on the individual. Governments, international organizations have a role to play. The question of this panel is, can democracy survive AI? I'm with you, Matt. I think it can, and I have three reasons for that.

Nik Gowing: And you think it's under threat?

Michaela Küchler: It's under threat, but I think it can survive. First of all, my kids. When they I have three kids aged 18 to 35, 34. And when they scroll through TikTok, I say, well, you're scrolling so quickly, why aren't you stopping? Oh, Mama, this is nothing. This is just created by AI. This is nothing I need to see. So one, this is more anecdotal. Secondly, democracies have survived technological and social change, which was fundamental before. And thirdly, a great deal will depend on us and on us now. And I see that AI is developing fast, and the problem is the time. And democracies usually are slow in reacting because they have a process of consulting and getting everybody heard. But the time to act is now is yesterday.

Nik Gowing: Renata, does that correspond to your reflections? You have a minute, please.

Renata Salecl: Yeah. I'm more pessimistic here until, you know, we stop kind of allowing big corporations to be pretty much totally in charge of the development of AI until we know what algorithms actually are doing, have access to study them, until there is the end of this monetarization, techno-feudalism, I would say. Then we cannot say that there is a democratic oversight of where AI is going. About the weapons, I would say that the war in Gaza has shown us, you know, the use of AI has been absolutely devastating for the civil population. The genocide in Gaza really turned to the AI, the lavender, you know, the gospel, where is daddy, all these machineries actually had a human in the loop. Human had 20 seconds maybe to decide among maybe 500, you know, kind of potential targets. And there was a collateral damage in, you know, the very algorithm, you know, kind of operating it. So this is my huge fear. And what is happening? Ukraine, the first drone which was recently discovered that maybe have decided on its own, you know, who might be the target.

Nik Gowing: Latch, can I just ask you, I'm sitting here thinking, how does the Pope govern? Does he do it by iPhone? Does he do it by laptop? Does he do it by AI? Help us understand how the church is changing. And I'm going to give you a minute only. But you've created that image, that impression that something big is changing in a traditional institution like yours.

Mihăiţă Blaj: The Pope is using his iPhone, his smartwatch. The Pope is chatting, he's having conversation with us.

Nik Gowing: Is he on WhatsApp?

Mihăiţă Blaj: Yes. But he is governing. He is governing through his everyday relationship with his collaborators using those instruments. So I'm trying to be informed, trying to listen as much as he can, trying to use his brain, and using the good instruments that we have. So I think that's the point. I would add a concept.

Nik Gowing: So he's communicating directly with his flock at every level?

Mihăiţă Blaj: At every level, it's impossible because we are 1 billion 300 million, so.

Nik Gowing: But you know what I mean.

Mihăiţă Blaj: I would add a concept to what we said. That is true education, regulation, optimism, et cetera. I would add shared responsibility. I was impressed to hear important VIPs from high tech coming up and saying, look, there is a danger here. There is a potential threat here. There is a problem here. That's honest. That's responsible. I was impressed to hear you that we said there is a threat here. This CIO is behaving wrong, et cetera. So we need to build shared responsibility at every level: workers, engineers, governors, et cetera.

Nik Gowing: Alexander, I want to give you the last word because you're doing some important work. And there was a video which we weren't going to show, which is a bit out of date now. But what work are you doing on this at RUC?

Oleksandr Danylyuk: Yeah. So I think that what is important to mention is our new book, which was authored by myself and my friend Jack Wottling. It's Subvert the State: Ecosystem of Russian Active Measures. It's available on Amazon. And it's about different things. And it's about how actually because it's mostly about strategic influence operations. So it's how Russians can actually use this natural stupidity more efficiently, also by using artificial intelligence. Not just about this, but yeah, cognitive warfare is about people's psychology, not about artificial intelligence, finally. And I personally believe that actually our understanding of these threats and again, I think that technologies are not threats. So like badwill actors, they are. And that's why our permanent center, you know, to actually to analyze Russian intelligence services is very important because they can use artificial intelligence today. They can use something different tomorrow. But we together need to understand that actually all of these technologies are used by human beings. And this year, the center is open just for some selected NATO countries, but the next year, we hope actually to invite more, including Slovenia, most probably some others. So let's stay in touch and let's do it together.