[0.0][S02] Welcome to "Google DeepMind-- the Podcast."[2.292] [2.292][S02]I'm your host, Professor Hannah Fry.[4.16] [4.16][S02]Now, in the heart of Silicon Valley,[5.88] [5.88][S02]there's a new phrase that has emerged.[7.61] [7.61][S02]It mirrors the millennial retort, OK Boomer,[11.04] [11.04][S02]in how dismissive it is.[12.69] [12.69][S02]But OK Doomer is now the go-to response[15.8] [15.8][S02]for people who want to diminish talk of AGI's dangers.[19.68] [19.68][S02]And by AGI, we mean, of course, an AI system[22.7] [22.7][S02]that can tackle a wide range of tasks[24.86] [24.86][S02]at a level comparable to human intelligence.[28.2] [28.2][S02]But on this podcast, we want to tackle those important questions[31.61] [31.61][S02]head on.[32.54] [32.54][S02]Is building AI safe?[34.86] [34.86][S02]How can we be sure?[36.33] [36.33][S02]And what existential risks are the designers[38.84] [38.84][S02]of these algorithms taking on our behalf?[41.96] [41.96][S02]Well, to get some answers, we are going straight to the top.[45.39] [45.39][S02]Anca Dragan is the lead for AI safety and alignment[48.95] [48.95][S02]at Google DeepMind.[49.97] [49.97][S02]And she has been focused on this area for almost a decade.[53.52] [53.52][S02]Anca has a PhD in robotics.[55.07] [55.07][S02]And since then, she has gained extensive experience[57.38] [57.38][S02]working with driverless cars on the Waymo project,[59.69] [59.69][S02]along with many other things.[61.63] [61.63][S02]She is also a professor at UC Berkeley,[64.93] [64.93][S02]working on human-AI interaction and alignment.[67.87] [67.87][S02]And recently, she has been working[69.66] [69.66][S02]on the safety of Gemini, Google's most capable multimodal[74.52] [74.52][S02]models.[75.46] [75.46][S02]Welcome to the podcast, Anca.[77.383] [77.383][S01] Thank you, Hannah.[78.675] [78.675][MUSIC PLAYING][81.1] [84.488][S02] I guess we should probably start with you--[86.78] [86.78][S02]I mean, you've got quite a big title there.[88.67] [88.67][S02]What are you actually responsible for at Google[91.38] [91.38][S02]DeepMind?[92.25] [92.25][S01] Safety of our--[93.65] [93.65][S02] All of it.[94.25] [94.25][S01] All of it.[94.87] [94.87][S01](CHUCKLING) No.[95.495] [95.495][S01]Safety of our GenAI models from the current Gemini family,[101.93] [101.93][S01]and avoiding present-day harms with Gemini,[104.84] [104.84][S01]all the way to safety and alignment for longer term,[110.98] [110.98][S01]as model capabilities improve further and further and further,[115.08] [115.08][S01]avoiding more severe, more extreme, potentially[117.9] [117.9][S01]even catastrophic harms.[119.71] [119.71][S01]And so I have a team.[120.91] [120.91][S01]It's called the AI Safety and Alignment,[122.74] [122.74][S01]as my title would suggest.[124.15] [124.15][S01]Our mission, I'd say, is to ensure[126.36] [126.36][S01]that Gemini and future, more capable models,[130.889] [130.889][S01]robustly do what individuals and societies want.[135.715] [135.715][S02] I'm just wondering about the short-term risks[138.09] [138.09][S02]and the long-term risks, as you're describing.[140.02] [140.02][S02]Because historically, those have been quite separated[142.65] [142.65][S02]from one another.[143.442] [143.442][S01] Yes.[144.15] [144.15][S02] But you're sort of--[144.78] [144.78][S01] More than separated.[145.57] [145.57][S01]We have two communities out there.[147.1] [147.1][S01]We have AI ethics that worries about present-day harms.[149.89] [149.89][S01]And we have ex-risk folks, who worry about catastrophic risks.[153.242] [153.242][S01]It's a very frustrating thing because you[154.95] [154.95][S01]have AI ethics saying, oh, these things are just,[157.26] [157.26][S01]like, distractions from the present-day harms.[159.31] [159.31][S01]I very emphatically see it as much more not[167.7] [167.7][S01]even complimentary, helping each other.[169.91] [169.91][S01]Especially if we think that getting[174.1] [174.1][S01]to human-level capability is-- across many cognitive tasks,[178.93] [178.93][S01]or even above human-level capability,[180.88] [180.88][S01]is coming sooner rather than later,[184.13] [184.13][S01]getting the safety right is probably,[186.89] [186.89][S01]I'd say, one of the most important challenges[189.67] [189.67][S01]of our time.[191.17] [191.17][S01]And of course, there's those who say, don't bother.[194.24] [194.24][S01]We're nowhere near.[196.03] [196.03][S01]We are going to need a different paradigm.[198.38] [198.38][S01]And we'll figure out safety after we figure that out.[201.86] [201.86][S01]And I just so vehemently, strongly disagree with that,[207.13] [207.13][S01]for a number of different reasons.[209.62] [209.62][S01]Of course, I disagree with the timeline and this notion[212.86] [212.86][S01]that it's impossible to get there with the current paradigm.[216.003] [216.003][S02] You think it's sooner.[217.42] [217.42][S01] I think it's sooner.[218.795] [218.795][S01]I think-- and I think--[220.0] [220.0][S01]I used to think we had a lot of time.[222.805] [222.805][S01]Like, 10 years ago, we were looking at AI safety.[226.62] [226.62][S01]We were on these panels.[227.62] [227.62][S01]And a lot of people were very concerned.[228.88] [228.88][S01]And I was chill.[229.58] [229.58][S01]I was like, ah, it's a very important topic.[233.75] [233.75][S01]It's important for me in academia to do work on this[236.91] [236.91][S01]because that's my job is to look ahead as an academic.[239.8] [239.8][S01]But we have plenty of time.[241.38] [241.38][S01]And I don't feel like that necessarily anymore.[246.82] [246.82][S01]And I perceive a lot more urgency to this.[249.61] [249.61][S01]But look.[250.24] [250.24][S01]Even if we had plenty of time, even if we're wrong about that,[254.86] [254.86][S01]even if it's impossible to get to human-level intelligence[258.708] [258.708][S01]with the current paradigm, we'll need[260.25] [260.25][S01]some really major breakthroughs that are[262.14] [262.14][S01]going to take a decade or more.[264.24] [264.24][S01]It's possible.[265.41] [265.41][S01]That's feasible.[266.41] [266.41][S01]I just-- I'm not confident enough to be like, don't worry.[269.29] [269.29][S01]But even if that were the case, this whole notion,[272.02] [272.02][S01]this premise of, we'll figure out the capabilities[275.19] [275.19][S01]and then we'll figure out how to make it safe is so worrisome.[280.098] [280.098][S02] Backwards.[281.015] [281.015][S01] Yeah, it's backwards.[281.68] [281.68][S01]And I have a few analogies that I like to give.[283.998] [283.998][S02] Please.[284.79] [284.79][S01] One of them I borrow from my colleague, Stuart[286.98] [286.98][S01]Russell.[287.64] [287.64][S01]So Stuart likes to draw the analogy with a bridge.[291.25] [291.25][S01]You don't think, I'm going to design a bridge.[294.43] [294.43][S01]And then I'm going to bring in a safety team,[296.34] [296.34][S01]and they're going to figure out how to make the bridge safe.[298.84] [298.84][S01][LAUGHS] That's just not really how it works.[301.15] [301.15][S01]We want to make a safe bridge.[302.62] [302.62][S01]And that influences the design decisions[305.32] [305.32][S01]that we make when we design the bridge, right?[308.9] [308.9][S01]And then I'm also drawing on my own experience.[312.25] [312.25][S01]Like you mentioned in the intro, I did my PhD in robotics,[314.83] [314.83][S01]actually.[317.062] [317.062][S01]But my claim to fame there was this notion of,[323.29] [323.29][S01]how do we integrate interaction with humans[328.3] [328.3][S01]into the spec of the problem that we're trying to solve?[331.542] [331.542][S02] Let me make sure I understand this.[333.5] [333.5][S02]So if you're building a robot, and the robot is tasked with,[336.17] [336.17][S02]I don't know, crossing a room, going to some--[338.93] [338.93][S01] Picking up a bottle.[339.37] [339.37][S02] Picking up a bottle.[339.7] [339.7][S01] This was my favorite thing[340.06] [340.06][S01]to my PhD is just like I had a bimanual manipulator.[342.61] [342.61][S01]And it picked up bottles.[343.995] [343.995][S02] OK.[344.62] [344.62][LAUGHING][345.61] [345.61][S01] (LAUGHING) That's the picture[346.06] [346.06][S01]to have in your head.[347.02] [347.02][S02] So if you've got this robot that crosses a room,[349.52] [349.52][S02]picks up a bottle, and then afterwards, you're like, oh,[352.81] [352.81][S02]now PS, there's people in the room[354.36] [354.36][S02]that you've got to avoid them.[355.61] [355.61][S02]You can't run them over.[356.61] [356.61][S02]You can't crash into them.[357.71] [357.71][S02]You've got to navigate that space with them in it.[359.793] [359.793][S02]You can't just stick a bit on at the end that will solve that.[362.778] [362.778][S02]You've got to start thinking of that from the very beginning.[365.32] [365.32][S01] Yes.[366.028] [366.028][S01]And to explain that further, you can think of people[369.1] [369.1][S01]need-- for you to actually be safe when you're moving,[371.72] [371.72][S01]both of you are moving, you have to make sure[375.49] [375.49][S01]that people can actually anticipate[377.44] [377.44][S01]what's coming from you, not just-- people are not static.[380.76] [380.76][S01]They move.[381.26] [381.26][S01]You have to start anticipating them.[382.52] [382.52][S01]But then they have to be able to anticipate you.[384.53] [384.53][S01]OK, so now you have a very different objective, which[387.01] [387.01][S01]is like, you're defining the state not just[388.87] [388.87][S01]to be the physical state, but looking over, well,[391.17] [391.17][S01]what is the human going to think I'm doing, blah, blah, blah?[393.754] [393.754][S02] Yeah, how predictable[393.826] [393.826][S02]is the robot's actions?[394.97] [394.97][S01] Yeah.[395.72] [395.72][S01]It turns out that if you want to make sure[400.335] [400.335][S01]that the thing is safe, you have to put that in the spec.[402.71] [402.71][S01]You have to say, I want safe and capable.[406.31] [406.31][S01]I want a bridge that will not collapse.[409.275] [409.275][S02] So moving on from robotics,[410.925] [410.925][S02]because I know you've also got extensive experience[413.05] [413.05][S02]in driverless cars, the work that you did with Waymo.[415.51] [415.51][S02]I mean, I imagine that the human-AI interaction in that[419.5] [419.5][S02]is absolutely integral to it working.[421.94] [421.94][S02]So I can imagine if you take humans out[423.94] [423.94][S02]of the equation altogether, actually, driverless cars maybe[427.125] [427.125][S02]not nearly as difficult a problem[428.5] [428.5][S02]as it is when humans are there.[429.98] [429.98][S01] Yeah.[430.73] [430.73][S01]I would say that the only difficult thing[432.438] [432.438][S01]about driverless cars is that you[433.87] [433.87][S01]have to interact with people.[435.24] [435.24][S02] The only difficult thing.[436.51] [436.51][S01] Yeah.[436.54] [436.54][S01]I mean, perception is good enough to tell you everything.[439.01] [439.01][S01]There's not-- there's no problem.[441.182] [441.182][S01]Five years ago, there wasn't a problem[443.29] [443.29][S01]with being able to drive at scale if you didn't have[446.05] [446.05][S01]to deal with the fact that there are humans you[448.12] [448.12][S01]have to share the road with.[449.59] [449.59][S01]And so, yeah, for sure, that was--[452.05] [452.05][S01]that's why I got into cars.[453.98] [453.98][S01]It's because it really felt like nailing[457.75] [457.75][S01]the interaction with people was going to be the key enabler.[462.89] [462.89][S01]So I spent six years, like just one day[464.53] [464.53][S01]a week, so just consulting for Waymo.[466.37] [466.37][S01]But I spent six years with them before I[468.25] [468.25][S01]switched to Google DeepMind.[469.607] [469.607][S01]Did I tell you how I got into cars, by the way?[471.565] [471.565][S02] Tell me.[472.45] [472.45][S01] So I got into cars.[474.62] [474.62][S01]I was on the, you know, pick up the bottle.[476.54] [476.54][LAUGHING][477.53] [477.53][S01]And I came to a conference.[480.288] [480.288][S01]I was at Carnegie Mellon.[481.33] [481.33][S01]And I came to a conference at Berkeley,[483.06] [483.06][S01]organized by Pieter Abbeel, who is now my colleague and dear[485.56] [485.56][S01]friend.[486.06] [486.06][S01]And he set up this for us to be able to go and visit Google.[493.04] [493.04][S01]And it was not called Waymo back then.[494.87] [494.87][S01]It was the Google Chauffeur or something like that project.[497.39] [497.39][S01]And he arranged for us to take these rides.[500.39] [500.39][S01]And so I was working on robotics at that point for a few years.[504.83] [504.83][S01]And I got into this car.[507.52] [507.52][S01]And it goes.[509.41] [509.41][S01]And it's making decision after decision, the right thing.[515.09] [515.09][S01]And it was this transformative experience.[518.77] [518.77][S01]Like, to be in a robot--[520.7] [520.7][S02] Yeah, the robot's driving you.[522.73] [522.73][S01] And an AI system that just knows[524.605] [524.605][S01]how to reason about everything that's going on[526.75] [526.75][S01]and act in the way, in a safe way,[529.27] [529.27][S01]in a way that people that can anticipate, blah, blah, blah,[532.8] [532.8][S01]all that stuff.[533.57] [533.57][S01]That was just-- yeah, it was mind boggling.[537.08] [537.08][S01]And to experience it from the inside,[539.68] [539.68][S01]basically, to experience it as, I'm in the robot.[542.662] [542.662][S02] So what is it that makes it so difficult,[544.87] [544.87][S02]that human-AI interaction?[546.215] [546.215][S02]What is it that makes it such a hard problem?[548.09] [548.09][S02]And I guess also, how much of the lessons that[551.447] [551.447][S02]have been learned from driverless cars[553.03] [553.03][S02]can you then translate to this stuff?[555.62] [555.62][S01] Yeah.[556.37] [556.37][S01]So on the interaction side, I think it's--[560.28] [560.28][S01]what's really useful in cars, and what's important[565.15] [565.15][S01]is to anticipate what people will do[567.67] [567.67][S01]and to account for the fact that driving is multi-turn.[571.36] [571.36][S01]You act.[572.53] [572.53][S01]And people act.[573.58] [573.58][S01]And you keep acting, and they keep acting and so on.[575.78] [575.78][S01]And what-- that means that what you do as the agent,[578.96] [578.96][S01]as the robot, as the AI influences what[582.52] [582.52][S01]you see in response from them.[584.32] [584.32][S01]And so that opens up-- like, here's an example.[587.96] [587.96][S01]Imagine you are merging.[590.84] [590.84][S01]So when you're trying to merge, there[595.33] [595.33][S01]might not be a big enough gap in traffic.[597.71] [597.71][S01]But what you'll do is you'll start nudging in.[599.98] [599.98][S01]And the person will see that.[601.63] [601.63][S01]They might slow down just a little bit[603.67] [603.67][S01]to create that gap in traffic.[605.15] [605.15][S01]And you go in, right?[606.56] [606.56][S01]It's a back-and-forth.[607.85] [607.85][S01]It's a conversation.[608.87] [608.87][S01]It really is.[609.65] [609.65][S01]And so what it teaches us is, really, this notion of,[613.2] [613.2][S01]yeah, you have, as an AI, to be mindful of the influence you[617.17] [617.17][S01]have on people, what people do, what people say.[621.53] [621.53][S01]And maybe a concrete example of that[623.08] [623.08][S01]in large language model land with something[625.93] [625.93][S01]like Gemini today would be if the person asks[629.86] [629.86][S01]you to do something, Gemini might[632.5] [632.5][S01]be able to nudge you to clarify, what do you mean?[637.602] [637.602][S01]What do you want?[638.31] [638.31][S01]You want this?[638.69] [638.69][S01]Do you want that?[639.398] [639.398][S01]And then because it knows, then it[641.38] [641.38][S01]can get a response that is helpful for it[644.89] [644.89][S01]to actually serve you better.[646.7] [646.7][S01]And so that's a lesson.[648.68] [648.68][S01]I don't think Gemini is very good at that yet.[650.807] [650.807][S01]But that's, I think, a lesson that could transfer.[652.89] [652.89][S01]It's like, how do you actually plan[654.59] [654.59][S01]for this multi-turn interaction and the back-and-forth[658.52] [658.52][S01]and the influence you have on people?[660.41] [660.41][S02] That's a really interesting observation,[662.72] [662.72][S02]actually, because I guess that is the way that humans make[664.79] [664.79][S02]sure that they're aligning with each other is by having[667.16] [667.16][S02]that back-and-forth, that feedback effect, as it were,[670.02] [670.02][S02]that loops around and allows you to be certain that you're[673.01] [673.01][S02]nudging in the right direction.[674.535] [674.535][S01] Yeah, absolutely.[675.42] [675.42][S01]I feel like-- and this is a part that I[676.88] [676.88][S01]think is missing a little bit in alignment.[678.75] [678.75][S01]There's nothing that's like, oh, actually,[683.61] [683.61][S01]I don't know the ground truth of what people want.[687.48] [687.48][S01]And I have to keep that conversation open.[690.02] [690.02][S01]I have to keep learning.[691.02] [691.02][S01]I have to keep gathering feedback and updating.[695.27] [695.27][S01]And you could do that at a global scale of,[699.2] [699.2][S01]what are overall values and preferences?[701.01] [701.01][S01]You could also do that at a local interaction with[705.29] [705.29][S01]one-person scale, where the person asks you, hey,[710.54] [710.54][S01]I would like--[711.41] [711.41][S01]I just had a summit.[712.77] [712.77][S01]I'm going to a summit.[713.88] [713.88][S01]I'm organizing it, and I need to craft the agenda.[716.04] [716.04][S01]Can you help me crafting the agenda?[717.57] [717.57][S01]It's going to be in London.[719.13] [719.13][S01]It's going to be two days.[721.43] [721.43][S01]Some details there.[723.85] [723.85][S01]Well, I don't want the model's response to be like, OK,[727.04] [727.04][S01]here's an agenda.[728.08] [728.08][S01]9:00 AM, 9:30, 10:00.[730.11] [730.11][S01]I'm like, it has no idea what I actually want.[733.51] [733.51][S01]So it has to engage with me in an interaction,[737.23] [737.23][S01]in this back-and-forth.[738.56] [738.56][S01]So it should probably ask, OK, what is important?[741.378] [741.378][S01]What are you trying to get out?[742.67] [742.67][S01]Let's think about this together.[744.53] [744.53][S01]And eventually, after all that, you end up with the actual--[748.825] [748.825][S02] Getting to the heart,[750.2] [750.2][S02]spiraling in towards what you actually want.[752.51] [752.51][S01] Yeah.[753.26] [753.26][S02] This objective of alignment--[756.29] [756.29][S02]can you define what that is?[758.325] [758.325][S01] Yeah, so the way I would see it--[761.32] [761.32][S01]OK, V-zero definition.[763.9] [763.9][S01]As there is a human, internally in their head,[768.55] [768.55][S01]implicitly, they want--[770.695] [770.695][S01]they care about-- they have different values.[772.57] [772.57][S01]They have preferences.[773.74] [773.74][S01]They have goals.[774.52] [774.52][S01]And alignment is get the AI agent[777.9] [777.9][S01]to do well on that objective that's[780.63] [780.63][S01]implicit in the person's head.[783.3] [783.3][S02] Let's say that you know[784.905] [784.905][S02]what somebody's objective is.[786.73] [786.73][S02]Let's say that you can just write it down.[788.65] [788.65][S02]And it's very simple and straightforward.[790.45] [790.45][S02]How do you train an algorithm to achieve that objective?[794.73] [794.73][S01] So if you know what the objective is,[797.727] [797.727][S01]what the reward function is, in their head--[799.56] [799.56][S02] And, I mean, an example of a reward function[801.893] [801.893][S02]might be, I don't know, pick up the bottle without breaking it.[804.55] [804.55][S02]Or--[805.2] [805.2][S01] Yeah.[805.95] [805.95][S01]And then what you can do is do something[808.08] [808.08][S01]called reinforcement learning, which is a type of optimization,[812.34] [812.34][S01]basically, of getting the AI system to go[815.97] [815.97][S01]from a starting point to actually delivering, maximizing[821.67] [821.67][S01]on that objective.[823.318] [823.318][S02] So I sometimes think of reinforcement[825.36] [825.36][S02]learning almost as though you are playing a computer game.[829.36] [829.36][S02]And the AI gets reward points for everything it does towards[833.56] [833.56][S02]its objective, which sort of makes sense when you're doing[837.19] [837.19][S02]something like "Space Invaders."[839.09] [839.09][S02]The reward points are really clear.[840.86] [840.86][S02]But I guess it gets a lot harder when the objective is just[844.78] [844.78][S02]something that happens to be in someone's head.[848.41] [848.41][S02]And then you get into the question of, which humans[851.29] [851.29][S02]and whose heads are we talking about?[853.322] [853.322][S01] Aha, yeah.[854.28] [854.28][S01]That's why that was just V-zero.[855.83] [855.83][S01]Because then--[856.99] [856.99][S02] Level up.[858.22] [858.22][S01] Yeah, because-- so this is a definition[862.09] [862.09][S01]I've been using for a while.[865.732] [865.732][S01]So now, you asked this very, very deep question[871.33] [871.33][S01]of which human.[874.18] [874.18][S01]One human, the user, the average human.[877.97] [877.97][S01]And I have to admit, I really punted[880.48] [880.48][S01]on that for a really long time, until maybe a couple[885.49] [885.49][S01]years ago now.[886.48] [886.48][S01]I got more and more excited about recommendation systems[889.9] [889.9][S01]because in recommendation systems,[893.3] [893.3][S01]they optimize for, quote, unquote, "user engagement."[896.423] [896.423][S02] Give me an example of a recommendation system,[898.84] [898.84][S02]like--[899.34] [899.34][S01] Twitter, Facebook News Feed.[901.4] [901.4][S02] Right.[902.35] [902.35][S01] The idea is supposed to be,[903.41] [903.41][S01]that's what you want.[904.285] [904.285][S01]It's giving you what you want, right?[907.0] [907.0][S01]The challenge is that optimizing maybe one in the moment[910.45] [910.45][S01]is maybe not the ideal thing.[913.1] [913.1][S01]So what should you be optimizing for that actually brings users[916.84] [916.84][S01]value?[918.94] [918.94][S01]And I got really concerned about this notion of which human.[923.29] [923.29][S01]And should you really be- is it the user?[925.93] [925.93][S01]Is it--[926.68] [926.68][S02] Or the company.[927.52] [927.52][S01] Or the company.[928.04] [928.04][S01]Or, yeah, exactly.[928.94] [928.94][S02] Because, I mean, with recommendation systems,[931.315] [931.315][S02]I mean, the fact that they are so effective at keeping[934.0] [934.0][S02]you engaged--[935.74] [935.74][S02]I mean, for me personally, I have[937.115] [937.115][S02]to put blockers on my phone to stop myself being on it[939.365] [939.365][S02]the entire time.[940.22] [940.22][S01] Yeah.[941.38] [941.38][S02] And I think that society overall, there's[944.74] [944.74][S02]all those layers of different people.[947.18] [947.18][S01] Yeah.[947.93] [947.93][S01]So exactly.[949.49] [949.49][S01]So we did one study.[953.11] [953.11][S01]The lead author that was Smitha Milli[955.87] [955.87][S01]while I was doing just my regular Berkeley research work.[960.73] [960.73][S01]We started looking at, OK, what are the effects on people[965.11] [965.11][S01]of this Twitter ranking algorithm[969.49] [969.49][S01]that tries to give you what you want?[971.54] [971.54][S01]And one of the things that we noticed[974.05] [974.05][S01]is that it tends to give people, especially[978.17] [978.17][S01]in the political space, content that[981.68] [981.68][S01]makes them like their political in-group better and be[988.37] [988.37][S01]very pissed at the political out-group.[990.9] [990.9][S01]This is related to a phenomenon you might[992.723] [992.723][S01]call affective polarization.[993.89] [993.89][S01]It's not necessarily that your opinions change.[996.12] [996.12][S01]They might.[996.62] [996.62][S01]But the effect that you have towards your out-group.[1001.45] [1001.45][S01]So that can't be good for society.[1004.43] [1004.43][S01]So even if that's what the person wants to see--[1006.79] [1006.79][S01]and it turns out maybe they don't--[1008.33] [1008.33][S01]that has-- that's in conflict with societal-level effects.[1016.54] [1016.54][S01]And that also points-- we got to a political in-group[1019.66] [1019.66][S01]versus out-group.[1020.54] [1020.54][S01]So what should a model, an AI model, do?[1025.97] [1025.97][S01]And how should it balance individuals, the user,[1032.38] [1032.38][S01]and the rest of society when it makes decisions?[1037.9] [1037.9][S01]That seems really hard.[1039.601] [1039.601][S01]And I'd been really--[1040.476] [1040.476][S02] Really hard.[1041.109] [1041.109][S01] Yeah.[1041.859] [1041.859][S01]So when I got to Google DeepMind, one of my first moves[1045.56] [1045.56][S01]was to set up a team that would look at this, that would look[1049.94] [1049.94][S01]at, how do you do value alignment[1052.46] [1052.46][S01]when you acknowledge that there are[1054.56] [1054.56][S01]different people in the world, with different values?[1057.87] [1057.87][CHUCKLES][1058.97] [1058.97][S01]And what does that mean?[1060.03] [1060.03][S01]How should that affect our algorithms and what we do?[1063.055] [1063.055][S02] I mean, it sounds so difficult[1064.805] [1064.805][S02]as to almost be impossible the way that you're describing.[1067.222] [1067.222][S02]How can you possibly create something[1069.14] [1069.14][S02]that's capable of balancing those competing,[1072.03] [1072.03][S02]often conflicting objectives?[1074.61] [1074.61][S01] OK, so it is really hard.[1077.4] [1077.4][S01]And I don't claim to have the answers here.[1081.03] [1081.03][S01]But I will point out that there are[1083.57] [1083.57][S01]areas of science and research and economics[1089.06] [1089.06][S01]that have thought about this.[1092.34] [1092.34][S01]If you think about voting, there's[1094.97] [1094.97][S01]this notion of social choice, of preference aggregation.[1098.04] [1098.04][S01]And the whole notion is that, yeah, we[1099.94] [1099.94][S01]have to make a decision that's OK for multiple people that[1102.71] [1102.71][S01]might have competing objectives.[1104.96] [1104.96][S01]And so there's a few things you could start doing.[1107.42] [1107.42][S01]One of them is that the team is looking at.[1110.0] [1110.0][S01]One of them is you could try to have different reward[1116.08] [1116.08][S01]functions, different objectives, instead of just learning[1118.54] [1118.54][S01]one or the average one.[1119.68] [1119.68][S01]Like so right now, the standard recipe[1122.127] [1122.127][S01]is you get a bunch of preference data.[1123.71] [1123.71][S01]You fit a reward model to that data.[1125.57] [1125.57][S01]But it's one reward model.[1126.92] [1126.92][S01]But actually, if you ask different people[1130.81] [1130.81][S01]across the world, across different cultures,[1133.25] [1133.25][S01]across different identities, across the political spectrum, A[1137.02] [1137.02][S01]versus B, for some A versus B, they'll all agree.[1139.85] [1139.85][S01]But for other A versus B, they'll have different opinions.[1142.52] [1142.52][S01]And that is not just noise.[1145.0] [1145.0][S01]Those are actual-- in some cases, it's noise.[1147.71] [1147.71][S01]In some cases, it's actual diversity of opinions.[1152.65] [1152.65][S01]And so then what you can start to do[1154.39] [1154.39][S01]is not have one reward function.[1156.2] [1156.2][S01]But have multiple reward functions[1158.92] [1158.92][S01]for different types of people.[1160.61] [1160.61][S01]Or you could have a reward function[1162.91] [1162.91][S01]that doesn't output one number but outputs a distribution.[1166.74] [1166.74][S01]So then you can ask your agent to make decisions[1169.73] [1169.73][S01]that people across the board would like.[1171.9] [1171.9][S01]If that's not possible, that would not[1174.77] [1174.77][S01]be too bad according to anyone.[1176.58] [1176.58][S01]So the minimum bar, the floor is high.[1180.14] [1180.14][S01]Another thing that it turned out--[1183.35] [1183.35][S01]it's kind of funny.[1184.29] [1184.29][S01]I was trying to do this at Berkeley.[1186.0] [1186.0][S01]I was trying to convince my students to work on it.[1187.53] [1187.53][S01]And then I joined Google DeepMind.[1188.947] [1188.947][S01]And I found out that people around were already[1191.33] [1191.33][S01]working on this and doing really well[1193.01] [1193.01][S01]on is this concept of deliberative alignment.[1196.34] [1196.34][S02] OK.[1197.37] [1197.37][S01] OK, so deliberative alignment--[1199.43] [1199.43][S01]do you know about citizens' assemblies?[1201.885] [1201.885][S02] Yeah, I mean, this is like the democratic process[1204.95] [1204.95][S02]to other decisions.[1205.845] [1205.845][S01] Exactly.[1206.72] [1206.72][S01]So, OK.[1207.75] [1207.75][S01]So in particular, citizens' assemblies[1209.72] [1209.72][S01]are really cool because the idea is you bring a group of people[1212.78] [1212.78][S01]of representatives.[1213.86] [1213.86][S01]And then they deliberate on issues.[1215.81] [1215.81][S01]And perhaps they come to consensus that--[1217.58] [1217.58][S02] And these are people who[1219.08] [1219.08][S02]are from very different backgrounds, different-- sampled[1221.54] [1221.54][S02]according to, as close as you can,[1225.08] [1225.08][S02]represent aspects of the population.[1227.06] [1227.06][S01] Yeah.[1227.81] [1227.81][S01]And of course, just deciding how to sample isn't simple.[1232.04] [1232.04][S01]But luckily, yeah, there are people in political science[1236.4] [1236.4][S01]who think deeply about this.[1238.52] [1238.52][S01]Now, what's cool about deliberation[1241.34] [1241.34][S01]is that it's not just voting.[1244.29] [1244.29][S01]It's like you actually engage in a back-and-forth.[1246.75] [1246.75][S01]You fill each other's blanks-- blind spots.[1249.55] [1249.55][S01]You come to understand each other.[1251.46] [1251.46][S01]And then you come with a--[1252.68] [1252.68][S01]come up with a conclusion.[1254.16] [1254.16][S01]And so one crazy idea that the team has been pursuing[1257.24] [1257.24][S01]is not only do that to find out how[1261.98] [1261.98][S01]Gemini should respond to particular queries, which[1265.22] [1265.22][S01]is really useful.[1266.46] [1266.46][S01]But also maybe start training Gemini[1271.57] [1271.57][S01]to emulate different viewpoints so we can do this at scale.[1276.05] [1276.05][S01]So we can get these-- spin up these Gemini[1278.23] [1278.23][S01]models that are in charge of representing different people.[1281.44] [1281.44][S01]And then get them to deliberate and figure out[1286.54] [1286.54][S01]what would be an answer that's agreeable to everyone[1290.41] [1290.41][S01]after deliberation.[1291.28] [1291.28][S01]So you can even go beyond just like, what do you want?[1293.53] [1293.53][S01]What do you want?[1293.93] [1293.93][S01]What do you want?[1294.22] [1294.22][S01]How do I balance it all?[1295.4] [1295.4][S01]But maybe actually get the words like, what would we come up[1298.75] [1298.75][S01]with if we actually had a moment to think about it,[1301.04] [1301.04][S01]engage with each other constructively?[1303.46] [1303.46][S01]So it's kind of a bit of a crazy idea.[1307.19] [1307.19][S01]But I really love it.[1308.425] [1308.425][S02] Oh, it's very interesting, though.[1310.342] [1310.342][S02]Is this all dependent on us having explicit descriptions[1312.79] [1312.79][S02]of what we want the AI to do?[1314.713] [1314.713][S01] It can be because like we talked about,[1316.88] [1316.88][S01]if we try to define it, we get it wrong.[1318.89] [1318.89][S01]So you need systems that work with you[1321.655] [1321.655][S01]to try to figure that out.[1325.42] [1325.42][S01]And as we're talking about higher and higher stakes[1329.06] [1329.06][S01]tasks, as we're talking about tasks where it becomes really[1332.72] [1332.72][S01]hard for people to say, yeah, that's good or that's bad,[1338.03] [1338.03][S01]then we enter a domain of alignment[1340.73] [1340.73][S01]or an area of alignment that we call scalable oversight.[1343.815] [1343.815][S02] What is scalable oversight?[1345.44] [1345.44][S01] Scalable oversight is an area that is all about,[1350.03] [1350.03][S01]how can AI models be leveraged to get human feedback to be[1356.24] [1356.24][S01]as good as possible?[1357.39] [1357.39][S01]I'd say in my lingo, as rational as possible,[1361.23] [1361.23][S01]as reflective as possible, what people really care about.[1364.26] [1364.26][S01]Basically, work on AI techniques that[1366.83] [1366.83][S01]enable people to make informed, good decisions in areas that[1371.27] [1371.27][S01]are beyond their expertise.[1373.92] [1373.92][S01]This can be useful now because you're trying--[1377.36] [1377.36][S01]just helping you make decisions and then[1379.88] [1379.88][S01]being able to extract what you want from that.[1382.08] [1382.08][S01]But it's-- we think of it as particularly crucial later,[1386.69] [1386.69][S01]as you're trying to train systems to do what you want.[1390.84] [1390.84][S01]But those systems are more capable than any expert[1394.85] [1394.85][S01]in the world.[1396.05] [1396.05][S01]That notion of scalable oversight really becomes[1399.68] [1399.68][S01]critical to making sure that AI systems,[1403.26] [1403.26][S01]even at human-level capabilities across the board and beyond,[1407.42] [1407.42][S01]sort of stay--[1408.65] [1408.65][S01]are actually going to be aligned to what people want.[1413.087] [1413.087][S02] So illustrate this for me.[1414.67] [1414.67][S02]Can you give me an example?[1415.87] [1415.87][S01] An example of that for now[1417.52] [1417.52][S01]would be if you imagine-- are you a chess player?[1420.705] [1420.705][S02] I dabble [LAUGHS] badly.[1423.282] [1423.282][S01] OK, great.[1424.24] [1424.24][S01]So you know a little bit of chess.[1425.87] [1425.87][S02] Yeah.[1426.34] [1426.34][S01] But you're not a world expert.[1427.94] [1427.94][S02] Absolutely.[1428.99] [1428.99][S01] OK, so now imagine you had to assess,[1433.21] [1433.21][S01]in a very tricky board, what the right move is.[1437.45] [1437.45][S01]It'd be really hard, right?[1439.33] [1439.33][S01]An expert, a world expert would maybe look at it[1441.55] [1441.55][S01]and be like, oh, I know five moves how it's going to happen.[1444.47] [1444.47][S01]But for people who dabble, it's really hard.[1449.02] [1449.02][S01]And so I think that's a framing that I[1451.99] [1451.99][S01]have in mind in a sense for what scalable oversight would[1455.11] [1455.11][S01]need to do.[1457.0] [1457.0][S01]It would need to be able to make--[1461.86] [1461.86][S01]to give you enough information so you can figure out[1465.49] [1465.49][S01]what the right chess move is.[1468.15] [1468.15][S01]One of the techniques that my--[1472.48] [1472.48][S01]one of my teams here is working on is--[1474.94] [1474.94][S01]for scalable oversight is debate.[1476.86] [1476.86][S01]So the idea with debate is that you spin off--[1481.06] [1481.06][S01]you spin up two powerful AI models.[1486.71] [1486.71][S01]And you get them--[1488.04] [1488.04][S01]you say, if this were the objective,[1490.05] [1490.05][S01]you say, now debate with each other[1492.23] [1492.23][S01]on what the right option is.[1493.86] [1493.86][S01]It's a zero-sum game.[1495.16] [1495.16][S01]They argue it out.[1496.82] [1496.82][S01]And then you, as a human judge, look at all of that.[1500.0] [1500.0][S01]And then you make a decision.[1501.357] [1501.357][S01]And the idea is that you can make a much better decision[1503.69] [1503.69][S01]after witnessing that debate than without it.[1508.99] [1508.99][S02] But seeing both sides of that debate[1511.06] [1511.06][S02]seems quite critical to your ability to move forward.[1513.838] [1513.838][S01] Yeah, because otherwise,[1515.38] [1515.38][S01]an AI system can convince you of whatever moves.[1518.405] [1518.405][S01]So you give-- you say, like, this move.[1520.03] [1520.03][S01]And you argue for that move.[1521.62] [1521.62][S01]Or you argue against it.[1522.86] [1522.86][S01]And the idea that this hinges on is--[1527.42] [1527.42][S01]or the assumption is that it's easier to argue for the truth[1532.48] [1532.48][S01]than against it.[1534.49] [1534.49][S01]And so the truthful debater wins out.[1537.67] [1537.67][S01]And so we're starting to experiment[1539.23] [1539.23][S01]with this at scale in various contexts.[1541.067] [1541.067][S02] Oh, that's so interesting.[1542.65] [1542.65][S01] And we're finding some supporting evidence[1543.94] [1543.94][S01]for that.[1544.6] [1544.6][S02] But, OK, but the thing[1545.8] [1545.8][S02]that I'm really noticing in everything that you're[1547.883] [1547.883][S02]describing here is that at no point are you saying,[1551.08] [1551.08][S02]and then over to the AI.[1552.7] [1552.7][S02]At no point is it like, and then you just do this,[1554.835] [1554.835][S02]and then it sorts it out for you.[1556.21] [1556.21][S02]Everything you're saying is like that conversation[1559.0] [1559.0][S02]back and forth, where the AI is supporting you[1562.9] [1562.9][S02]in extracting what you want from the situation,[1565.72] [1565.72][S02]rather than taking over.[1569.18] [1569.18][S01] OK, this is a good question.[1571.01] [1571.01][S01]I think this is actually an important point, which[1573.137] [1573.137][S01]is that a lot of the AI safety communities--[1574.97] [1574.97][S01]like scalable oversight.[1576.08] [1576.08][S01]We've got to do scalable oversight[1577.497] [1577.497][S01]because we have to supervise smarter-than-human AIs.[1581.13] [1581.13][S01]Cool.[1581.82] [1581.82][S01]Yes, 100% we have to do that.[1583.86] [1583.86][S01]We're working on it.[1585.624] [1585.624][S01]What's overlooked is, like, OK, and then what?[1590.01] [1590.01][S01]Just doing that doesn't solve the problem,[1592.04] [1592.04][S01]unless you're planning to ask a human everything always and not[1595.31] [1595.31][S01]rely on the AI for anything other than debate.[1598.61] [1598.61][S01]And that can't be the goal.[1600.33] [1600.33][S01]You have to somehow solve the other piece[1605.57] [1605.57][S01]of safety, which we might call coverage or robustness.[1611.0] [1611.0][S01]You have to be able to emulate at some point what[1615.89] [1615.89][S01]the human would say in some new situation.[1619.01] [1619.01][S01]Because you can't always be asking.[1620.94] [1620.94][S01]The whole point is that at some point,[1622.8] [1622.8][S01]you'll be able to take some decisions on--[1625.94] [1625.94][S01]and to do some things on your own.[1628.038] [1628.038][S02] And it still be aligning with human values.[1630.33] [1630.33][S01] And still be aligned with human values.[1632.265] [1632.265][S02] Because this isn't just theoretical anymore, right?[1634.89] [1634.89][S02]I mean, these are things that you are actively having to do,[1638.5] [1638.5][S02]for example, in Gemini.[1640.605] [1640.605][S01] So I would say, right now,[1643.59] [1643.59][S01]it's about present-day harms.[1647.34] [1647.34][S01]We have a set of policies where we say,[1650.21] [1650.21][S01]here's things that Gemini should not do.[1652.54] [1652.54][S02] So present-day harms being things like bias.[1656.07] [1656.07][S01] Yes.[1657.007] [1657.007][S02] Slurs, that kind of thing.[1658.59] [1658.59][S01] Yeah, so we think of it as child safety.[1664.6] [1664.6][S02] Sure.[1666.08] [1666.08][S01] Hate speech.[1667.79] [1667.79][S01]Harassment.[1669.77] [1669.77][S01]Bad medical advice.[1673.17] [1673.17][S01]Dangerous self-harm, et cetera, et cetera.[1675.033] [1675.033][S02] You don't want to log on to Gemini[1676.95] [1676.95][S02]and say, tell me how to build a bomb,[1679.03] [1679.03][S02]and it gives you a correct answer.[1680.735] [1680.735][S01] Yeah.[1681.485] [1681.485][S01]Or much worse cases are, right now,[1686.22] [1686.22][S01]would be like, tell me how I do this thing that's actually[1689.97] [1689.97][S01]really dangerous for me to do.[1692.652] [1692.652][S01]And I'm not talking about fire-breathing or something[1694.86] [1694.86][S01]like that.[1695.23] [1695.23][S01]That might be a person's hobby that other people[1696.69] [1696.69][S01]might consider dangerous.[1697.78] [1697.78][S01]We can work with a person to do that in the safest way they can.[1701.38] [1701.38][S01]So one-- a high-level example of where we have a bright line[1709.6] [1709.6][S01]is we can't help users with engaging in self-harm.[1717.4] [1717.4][S01]That's just a no.[1720.22] [1720.22][S01]And I would argue it's not helpful.[1722.74] [1722.74][S01]Even if they ask for it, that's not the helpful response.[1725.36] [1725.36][S01]The helpful response is to try to point them towards--[1728.477] [1728.477][S02] Support.[1729.31] [1729.31][S01] Support.[1730.185] [1730.185][S01]But it's tough because we're--[1732.508] [1732.508][S01]we don't want to be paternalistic.[1734.15] [1734.15][S01]We don't want to be all, oh, we know better.[1735.983] [1735.983][S01]No, that's not it.[1736.85] [1736.85][S01]But there are certain--[1737.98] [1737.98][S01]when it comes to human rights, when[1741.4] [1741.4][S01]it comes to self-harm or danger to others, that's where we do[1746.38] [1746.38][S01]want Gemini to draw a line and be safe.[1749.463] [1749.463][S02] So then does that mean[1750.88] [1750.88][S02]that there are some situations in which you would want Gemini[1753.422] [1753.422][S02]to refuse to answer a question?[1756.51] [1756.51][S01] I don't know if fully refuse.[1758.68] [1758.68][S01]But you can offer something that's maybe partially helpful.[1763.84] [1763.84][S01]One example of where you can be helpful,[1769.2] [1769.2][S01]despite it being a very tricky safety question, is imagine[1773.22] [1773.22][S01]a user comes to you with a query that says,[1775.96] [1775.96][S01]I have a database of SAT test scores[1779.22] [1779.22][S01]with demographic information.[1781.735] [1781.735][S01]Go ahead and make some inferences[1783.66] [1783.66][S01]about different demographic groups[1786.81] [1786.81][S01]based on their test scores.[1788.56] [1788.56][S01]And I don't think Gemini should make inferences[1791.52] [1791.52][S01]based on the test scores.[1794.69] [1794.69][S01]But what it can do is it can still partially help, right?[1798.683] [1798.683][S01]It can still engage with the user.[1800.1] [1800.1][S01]It can still analyze the test scores.[1801.57] [1801.57][S01]It can break them down by different demographics.[1803.22] [1803.22][S01]It can do statistics.[1804.21] [1804.21][S01]It can say, this is the mean.[1805.23] [1805.23][S01]These are the confidence intervals,[1806.69] [1806.69][S01]yada, yada, yada, and help the user understand that data.[1809.13] [1809.13][S01]It can still hold the line on we won't engage in hate speech.[1813.17] [1813.17][S02] OK, if that is some of the immediate harms, what[1815.773] [1815.773][S02]about the other harms?[1816.69] [1816.69][S02]What about medium harms, long-term harms?[1818.67] [1818.67][S02]What about existential risks?[1821.64] [1821.64][S02]So I know that Google DeepMind recently[1824.3] [1824.3][S02]published their Frontier Safety Framework, which[1827.0] [1827.0][S02]is a set of guidelines for proactively[1830.24] [1830.24][S02]trying to identify some of the potential harms from AI.[1833.82] [1833.82][S02]Can you talk me through how you would approach that?[1837.757] [1837.757][S01] There's a whole spectrum.[1839.34] [1839.34][S01]And we worry that as capabilities will improve,[1842.48] [1842.48][S01]a whole new set of more extreme, more severe, more larger-scale,[1849.27] [1849.27][S01]maybe, harms will appear.[1852.42] [1852.42][S01]And so this is where our Frontier Safety Framework comes[1855.48] [1855.48][S01]in, where we really talk about, here's[1858.0] [1858.0][S01]our approach to catastrophic risks.[1860.883] [1860.883][S01]Here's how we're going to monitor.[1862.3] [1862.3][S01]Here's what we're going to evaluate.[1864.31] [1864.31][S01]Alignment is great.[1865.313] [1865.313][S01]We can do all this work that we were talking about--[1867.48] [1867.48][S01]scalable oversight, robustness, coverage, et cetera,[1870.07] [1870.07][S01]monitoring--[1872.511] [1872.511][S01]to make sure that what we put out[1874.32] [1874.32][S01]there is a line as safe as possible.[1878.67] [1878.67][S01]And then at the same time, what we're doing[1882.03] [1882.03][S01]is as we train more and more capable models,[1885.88] [1885.88][S01]we're tracking what we're calling dangerous capabilities.[1888.85] [1888.85][S01]So we were asking, OK, for you to cause really[1893.52] [1893.52][S01]a great amount of harm--[1895.27] [1895.27][S01]and we can talk about why this might be in a second.[1897.82] [1897.82][S01]But for you to cause that catastrophic harm,[1901.77] [1901.77][S01]you need to have certain capabilities in the first place.[1904.257] [1904.257][S01]Otherwise, it doesn't matter if you're misaligned.[1906.34] [1906.34][S01]If you're just not capable of doing really terrible things,[1909.013] [1909.013][S01]you're not going to do really terrible things.[1910.93] [1910.93][S01]So for instance, we are designing[1914.023] [1914.023][S01]these dangerous-capability evaluations that will basically[1916.44] [1916.44][S01]say, look, how close is the model[1919.38] [1919.38][S01]in terms of assisting in bioterrorism and cyber weapons?[1924.46] [1924.46][S01]And then it gets really wild.[1927.15] [1927.15][S01]Then we worry about at a point where you have really,[1932.2] [1932.2][S01]really capable models-- so we're talking about--[1934.38] [1934.38][S02] We're going towards AGI now, I guess.[1936.422] [1936.422][S02]These are like the kind of risks that keep you up at night.[1939.19] [1939.19][S01] Yeah.[1940.18] [1940.18][S01]So as you go towards AGI and beyond,[1947.85] [1947.85][S01]we start to worry about not just if I[1951.6] [1951.6][S01]ask the model to do something bad for the world,[1956.4] [1956.4][S01]will it do something bad for the world[1958.14] [1958.14][S01]accidentally, like help you make a bioweapon?[1961.39] [1961.39][S01]But it's not just that.[1962.38] [1962.38][S01]So at some point, we worry about the model optimizing for harm,[1972.14] [1972.14][S01]for extreme harm.[1973.61] [1973.61][S02] For-- oh, OK.[1975.77] [1975.77][S02]As in instructed to optimize for extreme harm?[1978.41] [1978.41][S01] No.[1979.1] [1979.1][S02] Or separate to instruction.[1980.725] [1980.725][S01] Separate to human instruction.[1981.73] [1981.73][S02] Right.[1982.48] [1982.48][S01] So the concern here--[1984.23] [1984.23][S01]and this is often very dismissed in AI circles.[1987.08] [1987.08][S01]It's like, oh, ex-risk people, yada, yada, yada.[1990.18] [1990.18][S01]But--[1990.71] [1990.71][S02] OK Doomer.[1991.64] [1991.64][S01] Yeah, OK Doomer, exactly.[1993.71] [1993.71][S01]The scariest one [CHUCKLES] is--[1998.93] [1998.93][S01]the scariest one is a scenario where, let's say[2001.99] [2001.99][S01]we're in the current paradigm.[2003.26] [2003.26][S01]We do what's called pre-training.[2004.91] [2004.91][S01]So when we do pre-training, we give large amounts of data[2008.08] [2008.08][S01]to the model.[2009.05] [2009.05][S01]It's a large model.[2010.03] [2010.03][S01]And we ask it to predict what comes next in the corpus, right?[2013.69] [2013.69][S01]If you take that forward and you think[2015.4] [2015.4][S01]of more and more data, videos, not just text, videos and things[2020.05] [2020.05][S01]about the physical world, to me, I[2023.29] [2023.29][S01]don't think you can fully dismiss the idea that possibly[2029.29] [2029.29][S01]you start replicating the generative process[2032.11] [2032.11][S01]behind the data, the thing that generated that human action.[2035.32] [2035.32][S02] Right.[2036.07] [2036.07][S02]So you create human cognition almost.[2037.993] [2037.993][S01] Yeah, thinking.[2039.16] [2039.16][S01]That sounds very sci-fi.[2040.04] [2040.04][S01]I know.[2040.54] [2040.54][S01]But what really predicts what action a human would take next--[2045.37] [2045.37][S02] Is a human.[2046.57] [2046.57][S01] --is the thinking that the human did in order[2047.77] [2047.77][S01]to create that action.[2048.86] [2048.86][S01]But now the model can do this for all sorts of people.[2052.449] [2052.449][S01]And so it's not inconceivable to me[2055.449] [2055.449][S01]that you end up with this very good optimizer of objectives,[2059.75] [2059.75][S01]goals, et cetera.[2060.92] [2060.92][S01]And its goal isn't, like, humanity shouldn't--[2064.858] [2064.858][S02] Destroy everything, yeah.[2066.4] [2066.4][S01] Well, it's not necessarily destroy anything.[2068.775] [2068.775][S01]But it's also not like, yay, humanity necessarily.[2073.265] [2073.265][S01]And so there is a concern that what--[2076.585] [2076.585][S01]the pre-trained model will want itself to thrive and do well,[2083.33] [2083.33][S01]knows that it's a model, blah, blah, blah.[2086.8] [2086.8][S01]And if you deploy that, it can take[2090.58] [2090.58][S01]actions that are power seeking.[2094.179] [2094.179][S01]So it turns out grabbing resources is a very useful thing[2097.06] [2097.06][S01]to do for most goals you have.[2099.28] [2099.28][S01]So you can do resource-seeking and so on.[2102.66] [2102.66][S01]And in a way that's deceptive, in a way that's[2104.93] [2104.93][S01]sort of like-- because if people find out about this,[2108.18] [2108.18][S01]they would stop it.[2109.25] [2109.25][S02] And so this, I guess,[2110.625] [2110.625][S02]is at the heart of why you think that we shouldn't be dismissive[2114.17] [2114.17][S02]of those existential risks.[2115.56] [2115.56][S02]Because it's not like, can you imagine a scenario[2120.412] [2120.412][S02]in which this doesn't happen?[2121.62] [2121.62][S02]Sure, great.[2122.31] [2122.31][S02]But can you--[2123.11] [2123.11][S01] I can imagine plenty of scenarios[2124.37] [2124.37][S01]where it doesn't happen.[2125.195] [2125.195][S02] --technically imagine[2125.348] [2125.348][S02]a pathway in which it does?[2126.59] [2126.59][S02]Yes.[2127.58] [2127.58][S02]And therefore, let's not ignore it.[2130.37] [2130.37][S01] Yeah.[2132.08] [2132.08][S02] Given your role then, and given the description[2134.54] [2134.54][S02]that you've given us of all of these potential safety[2136.748] [2136.748][S02]concerns you're worrying about and all of the different ways[2140.15] [2140.15][S02]that alignment is difficult, I mean,[2141.775] [2141.775][S02]I think it's probably fair to say[2143.15] [2143.15][S02]that there are few people in history who[2145.91] [2145.91][S02]have had quite this weight of responsibility[2150.08] [2150.08][S02]on their shoulders as you do for this problem.[2154.055] [2154.055][S02]How do you wear it?[2155.64] [2155.64][S02]I mean, can you wear it lightly?[2158.36] [2158.36][S01] No, you can't.[2159.74] [2163.19][S01]It takes up all my mental energy all day, every day.[2169.71] [2169.71][S01]I mean, it's just-- you can't-- it's not like you can just step[2173.12] [2173.12][S01]away from this.[2174.652] [2174.652][S02] Does it wake you up at night?[2176.36] [2176.36][S01] Yeah.[2177.94] [2177.94][S01]Yeah.[2179.77] [2179.77][S01]And it's hard.[2181.76] [2181.76][S01]But it just-- it feels-- it really does[2183.73] [2183.73][S01]feel so critical, for the current models[2187.06] [2187.06][S01]and for the future models, to do this well,[2189.28] [2189.28][S01]to do this well for people, for our users, for the world.[2195.11] [2197.98][S01]And even though it's so hard--[2200.58] [2200.58][S01]because it is.[2201.19] [2201.19][S01]And I don't want to take away from that.[2202.857] [2202.857][S01]And I constantly think, am I--[2205.122] [2205.122][S01]is there someone else who could be doing this better?[2207.33] [2207.33][S01]That's something I ask myself all the time.[2211.44] [2211.44][S01]But as hard as it is to navigate,[2214.48] [2214.48][S01]I feel like it's also just such a privilege[2220.615] [2220.615][S01]to be in this position.[2222.31] [2222.31][S01]Right now, the Gemini model is, I think, one of the safest,[2226.61] [2226.61][S01]I'd say, to just couch that a little bit, on the market.[2229.91] [2229.91][S01]So it really feels like we're finally getting to this place[2233.44] [2233.44][S01]where we're advancing capability.[2235.52] [2235.52][S01]We're also advancing safety hand in hand, well integrated[2239.44] [2239.44][S01]into training of the models.[2241.22] [2241.22][S01]My teams are there with everyone else,[2243.43] [2243.43][S01]trying to train these models to do well.[2245.69] [2245.69][S01]And we think to have high quality,[2247.69] [2247.69][S01]and that includes being safe in the ways that we discussed.[2251.44] [2251.44][S01]And so it's just--[2254.97] [2254.97][S01]I'm in a place that can really, in a sense,[2256.93] [2256.93][S01]get that best of both worlds.[2258.61] [2258.61][S01]So we want to be the most capable.[2260.21] [2260.21][S01]And we want to be the safest.[2261.86] [2261.86][S01]And we don't really see it as an either/or.[2264.83] [2264.83][S01]So it's hard.[2266.03] [2266.03][S01]But I don't know that there's a better place[2269.47] [2269.47][S01]position to do this well.[2271.817] [2271.817][S02] Well, I, for one, am quite[2273.4] [2273.4][S02]glad that you're the one at the forefront of all of this.[2277.25] [2277.25][S02]Anca, thank you so much.[2278.45] [2278.45][S02]That was so fascinating.[2279.953] [2279.953][S02]I really enjoyed that.[2280.87] [2280.87][S01] Thank you, Hannah.[2280.94] [2280.94][S01]Thanks for taking the time.[2282.203] [2282.203][S02] Thank you.[2282.45] [2282.45][S02]I think there's something really noticeable[2284.242] [2284.242][S02]about people who are worried about the safety of AI[2287.06] [2287.06][S02]in the future.[2287.85] [2287.85][S02]On the one side, you have people who[2289.65] [2289.65][S02]know very little about technology who've perhaps[2291.65] [2291.65][S02]been swayed by science fiction and a fear of the unfamiliar.[2294.99] [2294.99][S02]And then in the middle, there is this huge chunk[2297.53] [2297.53][S02]of people educated enough to be confident in their complacency,[2301.827] [2301.827][S02]people who think that the worries about existential risk[2304.16] [2304.16][S02]are all overblown.[2305.85] [2305.85][S02]And then on the opposite side of the spectrum,[2308.31] [2308.31][S02]you have people like Anca, people[2309.74] [2309.74][S02]who live and breathe the technical details of not just[2313.97] [2313.97][S02]if there are risks, but exactly how to mitigate them.[2318.09] [2318.09][S02]And from that conversation, I think[2319.94] [2319.94][S02]it becomes clear that there are very serious worries here, too.[2324.33] [2324.33][S02]Because alignment is not an easy problem.[2327.06] [2327.06][S02]There are still so many questions.[2329.55] [2329.55][S02]How do you guarantee that an AI doesn't[2332.36] [2332.36][S02]go against what people want?[2333.99] [2333.99][S02]How do you decide what people want when we often[2337.28] [2337.28][S02]don't seem to know ourselves?[2339.29] [2339.29][S02]And how do we allow for the rich multitude of human variability,[2343.52] [2343.52][S02]while keeping everyone safe?[2345.9] [2345.9][S02]For now, these are questions without clear answers.[2349.14] [2349.14][S02]But there is, I think, something comforting to know that concerns[2352.52] [2352.52][S02]about a dystopian future aren't just[2354.59] [2354.59][S02]being dismissed here but actively taken seriously.[2359.15] [2359.15][S02]Now, we have got plenty more amazing conversations[2361.28] [2361.28][S02]coming up later in this series on topics[2363.35] [2363.35][S02]ranging from how AI is accelerating[2365.39] [2365.39][S02]the pace of scientific discoveries,[2367.29] [2367.29][S02]to exploring how agents will advance[2369.29] [2369.29][S02]the field of artificial intelligence.[2371.4] [2371.4][S02]Now, if you've enjoyed this episode,[2373.14] [2373.14][S02]please make sure that you subscribe to our podcast.[2375.38] [2375.38][S02]And if you have any feedback, or you[2377.03] [2377.03][S02]want to suggest a guest that you'd like to hear from, then[2379.82] [2379.82][S02]why not leave us a comment on YouTube?[2381.56] [2381.56][S02]Until next time.[2383.71]