15:40
We have this extraordinary moment where a potentially extremely disruptive technology emerges, and two countries-America and China-are completely dominant. And you know, all these tech bosses are saying that maybe there's a 10% chance of humanity going extinct. But these two "kings" negotiating with their "knights" do not seem to be very engaged in the discussion on the same level. You know, Trump was basically saying that we're happy with the way things are. Xi Jinping spoke in the spirit of maintaining human control over AI. Was this a signal of a completely different view, or was he simply taking the moral high ground, positioning himself as more responsible on this issue than Donald Trump?
I think it's still too early to judge this in general. What makes me happy is that this channel of communication has been opened. There was a major agreement to do this back in May. Now, as far as I understand, this channel is starting to work. A more important and fundamental question is whether there is a consensus on the level of urgency that advanced models currently represent. They are created both through closed models in the US and through open source or open-scale models in the case of China. I think China has come to the internal conclusion that its own cyber defenses are already robust, and that, in turn, stems from their view that AI has always been a problem in the context of unauthorized dissemination of information in a one-party
state, so they are masters of the firewall. You ain't seen nothing yet until we push our firewall in cyber defense against anything malicious that might arise from the autonomous actions of AI agents, let alone cases of their manipulation by third parties, unscrupulous actors, or rogue states. I believe this is a Chinese mistake; because even their cyber defense systems, in my opinion, are not reliable enough to cope with a massive attack on social, economic, or even military infrastructure, which under certain circumstances could be caused by the most advanced models if they were to go beyond human control. Doesn't it bother you that each side is retreating into its comfort zone, while something much larger is at stake, and neither side is currently
trying to deal with it In my opinion, neither the Chinese nor the American systems have yet reached a final internal consensus on this issue. I think there is a certain disconnect between the technology community in both countries, on the one hand, and the political community and policy experts on the other. In the case of China, you don't have to go far from Zhongnanhai towards the Chinese tech world to conclude that this is potentially very dangerous indeed. And in fact, the events that unfolded in technology itself between the May summit in Beijing and this September summit in Washington were happening with such speed and intensity that they caused real concern in the Chinese tech community. Can political and strategic What exactly are they
concerned about? They are concerned precisely because certain advanced models create agents capable of acting autonomously; and that, frankly, given the spread of these agents through the American or even Chinese system in other parts of the world, they could be developed further through diffusion and adaptation in such a way that it could be used against China itself. But you're a person who chooses your words very carefully, and you said you've heard this concern from big Chinese AI companies. However, when you've met with senior officials, are they focused on this to the point where you feel at ease, or are you a little worried that they're still thinking in terms of political security? As I said, I don't think they've reached a consensus,
as I see it in the American system, between technologists on one side and the political establishment on the other, to realize how real this threat is, how urgent it is, and what to do about it. I see these tensions in the Chinese system right now, and to some extent they are natural, as this is all developing very quickly. Because, as you rightly pointed out earlier, David, it challenges the historical view that we should only worry about one thing: the security of the regime and the unauthorized dissemination of information, which we have already blocked, and that's it. This is a problem of the American open society and open political system. Is there any real hope that these two countries will hold urgent talks while President Trump and
his administration talk about the need to maintain American dominance? Because in any other area you can think of, when you try to engage the Chinese, for example, nuclear arms control, the message is very clear: "As long as we don't have parity, as long as you have 10 times more warheads than we do, why should we even start these negotiations?" Because at the deepest level, the Chinese are convinced that the weaker party entering into negotiations will receive an unfavorable agreement imposed on it. So , as long as China is a catching-up AI power and the Americans are openly talking-whether it's Trump or actually Dario Amodei-about the need to keep the Chinese in second place, is that an absolute barrier to really meaningful negotiations, or do you
see them getting scared enough to agree to talk? China's default position is exactly the same as President Trump's position. This is an active strategic competition in AI across the entire stack, and whoever wins in AI wins everything. And no matter what polite modulations of Trump's statements the Chinese use, this is their working position. Anyone who is a realist and looks at the behavior of the Chinese system will come to this conclusion. Secondly, you are absolutely right to say that when they hear from the Americans, "Let's put some safeguards in place. Let's have a bilaterally agreed framework." And when this is reinforced by a recent article by Dario Amodei, which says that the strategic goal here is partly not just to protect themselves, but also to stay
ahead of the Chinese, the Chinese are not stupid, they read this and say, "Haha." This is exactly what we were thinking. Hence the massive reaction of the Chinese system to Dario's proposal, almost as intense as to President Trump. Yes. But there is one thing they agree on-that is, they all hate Dario Amodei. And so this is largely the default Chinese position. Not only are we in a competitive relationship, we are also not going to allow our hands to be tied on this issue. So here is the dilemma we all face due to the mutual distrust between the two systems, despite the personal chemistry that supposedly exists between the two leaders. Does a complete and utter catastrophe have to occur in the real world to provide the trigger
that will force Beijing and Washington to reach a moment of truth? And to say, "Oh my God, this is bigger and scarier than both of us, and we need to work together quickly." Or, if that's not enough, do we say we need a Caribbean crisis that nearly destroys humanity to bring us to that very moment of truth, or can we be a little more mature and actually realize from what the technologists are telling us that this is a real prospect, and it's much better to get ahead of events now. I am optimistic that, given China's commitment to science and technology- oriented leadership, they will soon come to the conclusion that this problem is significant, real, and urgent, and will be able to establish a negotiation process with President Trump, just as I am optimistic
about the ability of the American side to engage in such an engagement.
