[0.0][MUSIC PLAYING][2.97] [6.93][S01] Welcome back to "Google DeepMind, The Podcast."[9.66] [9.66][S01]I'm your host, Professor Hannah Fry.[11.77] [11.77][S01]Now, if you could build your own personal AI assistant,[15.82] [15.82][S01]what would it be like?[16.93] [16.93][S01]Would it be an efficient chief of staff[19.26] [19.26][S01]that helps you to make the most of every moment,[21.58] [21.58][S01]or maybe a digital twin who can attend[23.67] [23.67][S01]boring meetings on your behalf?[25.84] [25.84][S01]Well, it goes without saying that these scenarios are mostly[28.92] [28.92][S01]hypothetical, for now, but believe it or not,[31.45] [31.45][S01]there is someone at Google DeepMind whose actual job it[34.86] [34.86][S01]is to think through them all and all[37.56] [37.56][S01]of their attendant opportunities and dangers.[40.6] [40.6][S01]Iason Gabriel is a Staff Research Scientist[43.2] [43.2][S01]in Google DeepMind's Ethics team.[45.37] [45.37][S01]Before joining, he taught moral and political philosophy[48.57] [48.57][S01]at Oxford University and worked at the United Nations.[52.15] [52.15][S01]His work at the intersection of ethics[54.06] [54.06][S01]and artificial intelligence has earned him[56.13] [56.13][S01]recognition as one of the leading thinkers in the field.[59.27] [59.27][S01]In fact, he's just been featured in the Time 100 AI list.[63.49] [63.49][S01]Iason, welcome to the podcast.[65.08] [65.08][S02] Thank you.[66.122] [66.122][S02]Happy to be here.[67.085] [67.085][S01] I think it's probably[68.46] [68.46][S01]good to start with some definitions.[70.12] [70.12][S01]So what do you actually mean by an AI assistant?[73.86] [73.86][S02] So we're all familiar with generative AI[78.78] [78.78][S02]and things like ChatGPT and Gemini.[81.42] [81.42][S02]But there's this idea that these kind of base technologies[84.27] [84.27][S02]will become more and more capable down the line.[86.86] [86.86][S02]So there will be plugged into different kinds[89.85] [89.85][S02]of tools that will allow them to take action in the world.[93.45] [93.45][S02]We've seen them becoming more competent at reasoning.[96.605] [96.605][S02]And maybe, in due course, they'll[97.98] [97.98][S02]be able to pursue really complicated goals[100.2] [100.2][S02]rather than just producing very, very fluent text.[102.97] [102.97][S02]And so we have this idea of agents entering the world.[105.647] [105.647][S02]And then, the next question was, well,[107.23] [107.23][S02]what will the most useful or the most common form of an agent be?[112.95] [112.95][S02]And to us, it seemed that it was very[114.57] [114.57][S02]likely to be a kind of highly capable assistant.[117.44] [117.44][S02]Many of us would like to be plugged[120.15] [120.15][S02]into a technology that can take care of a lot of life tasks[123.51] [123.51][S02]on our behalf.[124.87] [124.87][S02]And so the assistant is a kind of agent,[127.28] [127.28][S02]but one that has a special relationship with the user,[130.37] [130.37][S02]which is that it's approximately tethered to the user's[133.21] [133.21][S02]intentions.[133.71] [133.71][S02]So it does what I tell it to do within reason[138.34] [138.34][S02]and potentially can help us on this life journey[141.67] [141.67][S02]in a variety of different ways.[143.165] [143.165][S01] What types of AI assistants[144.79] [144.79][S01]are we talking about here?[146.0] [146.0][S02] Yeah, so I think there's[147.64] [147.64][S02]different things that people have in mind when[149.86] [149.86][S02]they talk about assistants.[151.12] [151.12][S02]And they kind of range from the stuff that's[153.28] [153.28][S02]almost right in front of us to things that[155.68] [155.68][S02]are potentially really, really powerful and advanced[157.93] [157.93][S02]technologies.[159.58] [159.58][S02]So the thing that is right in front of us[162.43] [162.43][S02]is like the administrative assistants, interfaces[164.95] [164.95][S02]with your calendar, it can do your meetings for you.[168.01] [168.01][S02]Actually, some kinds of conversational chat bot[171.01] [171.01][S02]are already quite sophisticated.[172.9] [172.9][S02]And you can imagine having really good learning experiences[176.62] [176.62][S02]with an AI assistant that's designed[178.3] [178.3][S02]to help you master some skill.[181.36] [181.36][S02]But there are more capable things that could be built.[184.97] [184.97][S02]So sometimes it's a matter of taking[187.36] [187.36][S02]an example of what we have now and just extrapolating[189.7] [189.7][S02]into the future.[190.58] [190.58][S02]So we can imagine a kind of research helper.[193.4] [193.4][S02]But can we imagine a research helper[195.34] [195.34][S02]that's literally read every scientific paper in the world[198.28] [198.28][S02]and has a superhuman ability to synthesize information, right,[202.75] [202.75][S02]or even to generate novel hypotheses?[205.465] [205.465][S02]Then it becomes more of a thought partner.[207.71] [207.71][S02]And then there's kind of we have the administrative assistant.[210.59] [210.59][S02]But people sometimes think, well,[212.06] [212.06][S02]what is an administrative assistant[213.59] [213.59][S02]if it becomes really capable and it does tons of stuff,[216.34] [216.34][S02]it books your appointments for you,[218.44] [218.44][S02]finds schools for your kids to go to, looks up holidays.[222.65] [222.65][S02]They say, well, then it's like a chief of staff.[225.22] [225.22][S02]And this idea that I guess a chief of staff[228.13] [228.13][S02]is something that has a lot of executive capability.[231.18] [231.18][S02]So this idea, well, maybe if we could just[232.93] [232.93][S02]get that thing up and running, then we get all this free time[235.66] [235.66][S02]that we've been dreaming about.[237.94] [237.94][S02]And then, pushing towards the more long-term views, also[244.15] [244.15][S02]an idea that an AI could be a custodian of the self.[248.745] [248.745][S02]So that means that it's something[250.12] [250.12][S02]that you've presumably had these deep and meaningful[252.43] [252.43][S02]conversations with it and you're really[254.055] [254.055][S02]aligned in terms of where you want to get to and your goals,[257.42] [257.42][S02]and it helps keep you on track.[260.313] [260.313][S02]It's kind of a coach, but really,[261.88] [261.88][S02]something that as your life progresses,[265.69] [265.69][S02]you get more autonomy because it protects you[268.81] [268.81][S02]against certain kinds of distractions or mistakes[271.78] [271.78][S02]you might make.[272.9] [272.9][S02]And then, I think one final thing people[275.08] [275.08][S02]have started to talk about is a universal interface or a kind[278.83] [278.83][S02]of assistant that moves between different devices.[283.19] [283.19][S02]And so, maybe it does information retrieval here,[286.07] [286.07][S02]maybe it gives advice there, maybe it does these things here.[288.887] [288.887][S02]Of course, that really is a different world from the one[291.22] [291.22][S02]we're in now.[292.04] [292.04][S02]But according to some speculation,[295.51] [295.51][S02]that may be where we're heading.[297.05] [297.05][S01] OK, so paint me a kind[298.99] [298.99][S01]of utopic view of the future before we[300.585] [300.585][S01]get into some of the knotty philosophical questions[302.71] [302.71][S01]that this raises.[303.76] [303.76][S02] Yeah, absolutely.[304.33] [304.33][S01] So are we talking about, you, as an individual,[306.98] [306.98][S01]have your own personal version of this[309.58] [309.58][S01]or multiple different versions of it?[311.48] [311.48][S02] So that's an open question.[313.43] [313.43][S02]So there's a kind of--[315.34] [315.34][S02]there's a bit of an ongoing debate[317.32] [317.32][S02]about whether we would rather separate our assistant[320.38] [320.38][S02]or whether there could be a kind of universal assistant[322.9] [322.9][S02]that would be kind of-- it's like one assistant[325.54] [325.54][S02]to rule them all that does all of them.[328.24] [328.24][S02]One of the advantages of partitioning[330.31] [330.31][S02]is it's a little easier for us to handle psychologically.[333.11] [333.11][S02]And it may also be better from a privacy point of view.[337.72] [337.72][S02]I think the utopian version of it is kind of--[341.755] [341.755][S02]it's thing that we often don't get in practice,[343.97] [343.97][S02]but it's essentially that we get our time back.[346.51] [346.51][S02]So I think as adults living in the modern world,[350.0] [350.0][S02]we all feel encroachment.[351.2] [351.2][S02]It's like, oh, my gosh, if I could just[352.99] [352.99][S02]take care of that six hours of stuff that I need to do,[356.72] [356.72][S02]then I would be able to spend time with my friends,[359.84] [359.84][S02]enjoy some music, just hang out with the kids, or whatever.[363.47] [363.47][S02]And so I think the vision is that on one hand,[367.43] [367.43][S02]we'd get back a lot of time to do what's valuable for us.[371.75] [371.75][S02]And then, there's also this idea that,[375.07] [375.07][S02]through the ready availability of coaching,[379.61] [379.61][S02]educational input and things like[381.13] [381.13][S02]that, we might also be able to become more the people[384.25] [384.25][S02]that we want to be.[385.43] [385.43][S02]So we often have this other feeling, which is like, oh,[388.817] [388.817][S02]if only I had time to read those books[390.4] [390.4][S02]and learn that skill, et cetera.[392.41] [392.41][S02]But there's friction.[393.452] [393.452][S02]There's friction.[394.16] [394.16][S02]You have to sign up to a course, is[396.43] [396.43][S02]the person going to be a good teacher, you don't really know.[399.08] [399.08][S02]But if you have this kind of ready availability of knowledge[401.98] [401.98][S02]and trustworthy advice, that could also[404.08] [404.08][S02]be a massive resource.[405.863] [405.863][S01] I know you've got background[407.53] [407.53][S01]as a political theorist and an ethicist.[410.3] [410.3][S01]How did you end up working in AI?[412.21] [412.21][S02] I actually had two careers before this.[414.62] [414.62][S02]So I initially worked for the United Nations[417.07] [417.07][S02]in Sudan and Lebanon and was kind of motivated by,[422.08] [422.08][S02]as you would imagine, those kind of humanitarian concerns.[425.47] [425.47][S02]And then, after doing that for several years,[427.52] [427.52][S02]I decided to do a PhD in philosophy[429.31] [429.31][S02]and dig into these deeper questions[431.56] [431.56][S02]about global equality and global justice.[435.14] [435.14][S02]And I taught philosophy for a number of years.[437.78] [437.78][S02]But I had a sense that the action[440.68] [440.68][S02]was happening somewhere else.[442.58] [442.58][S02]And so I read a couple of books on AI.[445.81] [445.81][S02]For example, there's a beautiful book called "Automating[448.18] [448.18][S02]Inequality," and also, the famous book "Superintelligence."[452.75] [452.75][S02]And I just started to triangulate and think,[455.24] [455.24][S02]well, what if we have very, very powerful AI systems,[458.03] [458.03][S02]but they have bias in them, what if we have autonomous systems.[463.18] [463.18][S02]And so I did see that there was this kind[465.37] [465.37][S02]of coming wave of things that we needed to prepare for.[468.59] [468.59][S02]And just very fortunately, at that time,[471.02] [471.02][S02]I met the folks at DeepMind.[472.57] [472.57][S02]And they had also seen it coming, basically.[475.01] [475.01][S02]But that was more from a technological point of view.[477.74] [477.74][S02]There was the aspiration to build AGI.[481.36] [481.36][S02]And if you take that seriously, as with the agents,[485.6] [485.6][S02]it's like the implications are just truly profound.[488.24] [488.24][S02]So we got together and we've been working on it ever since.[492.05] [492.05][S01] But I suppose we should probably[494.23] [494.23][S01]talk about the paper that you have released.[496.918] [496.918][S01]You've been working on this paper for some time, right?[499.21] [499.21][S01]Tell me how it came about.[500.81] [500.81][S02] Yeah, so the paper was the result of,[503.74] [503.74][S02]I think at this point, almost a two year collaboration.[507.59] [507.59][S02]And we were really preoccupied with this question[510.07] [510.07][S02]of what comes after language models in the form[513.49] [513.49][S02]that we're familiar with.[515.12] [515.12][S02]And so I work on the ethics research team[518.14] [518.14][S02]within Google DeepMind.[519.87] [519.87][S02]And for us, this is a very high stakes question.[521.87] [521.87][S02]It's like, we really need to know what's[523.537] [523.537][S02]going to happen next so that we can understand what[525.76] [525.76][S02]the consequences might be, and then[527.53] [527.53][S02]reason backwards to making good decisions in the present moment.[530.75] [530.75][S02]We started to have this kind of dawning realization[534.31] [534.31][S02]that there would be an agentic turn[536.08] [536.08][S02]or that you could build all these things--[537.46] [537.46][S01] Agentic turn.[538.135] [538.135][S01]I like that.[538.69] [538.69][S02] Yeah, all these things[540.421] [540.421][S02]on top of language models.[542.312] [542.312][S02]And then, we just started to ask ourselves questions.[544.52] [544.52][S02]So it was this deck of, I think, 46 questions[547.18] [547.18][S02]that we came up with.[548.45] [548.45][S02]But a lot of them were like, well,[549.962] [549.962][S02]what happens then, what happens if you have a million[552.17] [552.17][S02]or a billion agents in the world.[554.57] [554.57][S02]That's quite a different society from the one we live in now.[557.81] [557.81][S02]And we gathered researchers from many different disciplines.[560.96] [560.96][S02]So we had economists, sociologists,[563.12] [563.12][S02]computer scientists, human computer interaction experts.[567.32] [567.32][S02]And along with Arianna Manzini and Geoff Keeling,[570.47] [570.47][S02]my co-authors, we pretty much asked everyone just[572.86] [572.86][S02]to tell us about their own expertise,[575.27] [575.27][S02]like, mapped onto this topic.[576.92] [576.92][S02]So we went and spoke to the privacy researchers.[579.34] [579.34][S02]And we said, what do you think privacy looks like in this world[582.43] [582.43][S02]where agents are interacting with one another on our behalf?[585.26] [585.26][S02]What do you think safety looks like in that world?[587.87] [587.87][S02]And so the paper.[589.375] [589.375][S02]it's a sum of, hopefully, quite a lot of wisdom[592.45] [592.45][S02]applied to one domain.[594.41] [594.41][S01] I'm just trying to think back as[596.35] [596.35][S01]to where we were two years ago.[598.43] [598.43][S01]So essentially, you're sort of saying,[600.61] [600.61][S01]we imagine that language models are[602.53] [602.53][S01]going to get to a point where they can execute[604.9] [604.9][S01]a series of tasks according to some objective[608.04] [608.04][S01]that you as an individual set.[609.55] [609.55][S01]What happens then, right?[610.883] [610.883][S02] Yeah, that's the starting thought, exactly.[613.3] [613.3][S01] But then, wasn't a lot of this quite speculative[615.96] [615.96][S01]then?[616.57] [616.57][S02] It was very speculative.[619.18] [619.18][S02]And I mean, we have a really elaborate way of describing it.[621.88] [621.88][S02]We say we do socio-technical foresight, which, I admit,[625.35] [625.35][S02]is not very catchy.[626.47] [626.47][S02]People aren't going to vibe with it.[628.86] [628.86][S02]But if we break it down, what it means[631.17] [631.17][S02]is we take something speculative,[633.57] [633.57][S02]and then the trick is to take it as a kind[636.27] [636.27][S02]of serious technical and sociological proposition[639.51] [639.51][S02]and be like, OK, but what do we really[641.31] [641.31][S02]know about sociology, what do we really know about the path[644.43] [644.43][S02]dependency of the technology, and can we[646.26] [646.26][S02]get the detail or high resolution on this vision that[650.1] [650.1][S02]allows us to do the kind of ethics analysis.[652.87] [652.87][S02]That's where you have to put in months and months of work.[655.82] [655.82][S01] I mean, this is a credible way[657.57] [657.57][S01]to go for all those science fiction questions that people[660.072] [660.072][S01]have floating around in their minds[661.53] [661.53][S01]and actually address them and start thinking about them[663.93] [663.93][S01]really seriously.[664.638] [664.638][S02] Yeah, yeah, yeah.[665.972] [665.972][S02]I mean, I think it's hopefully a kind[667.54] [667.54][S02]of template for the way we'll explore other topics.[670.612] [670.612][S02]And, of course, I mean, maybe we'll[672.07] [672.07][S02]talk about this at the end.[673.01] [673.01][S02]But the question is like, OK, what's[674.51] [674.51][S02]happening two years from now, right?[676.64] [676.64][S02]Where does our foresight need to reach to?[679.64] [679.64][S02]And it's one of the interesting things[681.79] [681.79][S02]about working for a technology company is you do have access[685.27] [685.27][S02]to privileged information.[686.93] [686.93][S02]But with it, you have this responsibility.[689.51] [689.51][S02]You have ability to potentially see some things a little bit[692.74] [692.74][S02]more clearly than folks who are not working in this context.[696.11] [696.11][S02]And so you really want to use that information, I mean, first[700.15] [700.15][S02]of all, to help the company make good decisions,[702.23] [702.23][S02]but really, to help the whole ecosystem prepare[705.34] [705.34][S02]for this credibly input into what's being built.[709.37] [709.37][S02]And so this was a kind of project that was really[712.03] [712.03][S02]aimed at everyone at the end of the day.[714.28] [714.28][S01] When you say "aimed at everybody,"[716.24] [716.24][S01]there are other people working in this space who you don't have[719.05] [719.05][S01]direct control over, right?[720.59] [720.59][S02] Yeah.[720.73] [720.73][S01] So how much power do you actually[722.605] [722.605][S01]have to stop these bad scenarios coming to be?[726.8] [726.8][S02] Yeah, I mean, that's a really good question.[729.5] [729.5][S02]It features a lot in--[731.32] [731.32][S02]there's a part of the study that looks at multi-agent scenarios.[735.92] [735.92][S02]So multi-agent scenarios are the ones[737.62] [737.62][S02]that arise when you have many different agents[740.86] [740.86][S02]and assistants deployed by many different actors.[744.05] [744.05][S02]And there's important questions like,[746.51] [746.51][S02]do they compete with each other.[748.22] [748.22][S02]So do they kind of battle it out?[750.5] [750.5][S02]And does that have a kind of stabilizing[753.16] [753.16][S02]or a destabilizing effect on society?[755.8] [755.8][S02]Or do they need to cooperate through some kind of interaction[758.68] [758.68][S02]protocol?[759.65] [759.65][S02]Or is this really a governance question?[761.62] [761.62][S02]Do we actually need new rules saying,[764.2] [764.2][S02]agents need a reliable identification,[766.715] [766.715][S02]we need to be able to trace who's agent[768.34] [768.34][S02]is doing what in the wild?[770.24] [770.24][S02]So I think this is the knowledge-based investigation[775.48] [775.48][S02]is the prelude to action.[778.01] [778.01][S02]And then, when we enter the world[779.56] [779.56][S02]of action, a lot of the things that we lay out,[782.6] [782.6][S02]these ethical issues, they're general issues.[787.16] [787.16][S02]And by sharing the information, we give other people[789.73] [789.73][S02]the opportunity to act as well.[792.25] [792.25][S02]Of course, we need to set a good example.[794.03] [794.03][S02]Google DeepMind is one of the major AI research[797.59] [797.59][S02]producers in the world.[799.96] [799.96][S02]And fortunately, we're entering this stage[802.31] [802.31][S02]where just building better AI means building ethical AI that[805.54] [805.54][S02]is good for the people who use it[807.91] [807.91][S02]and good for the society it enters into.[810.38] [810.38][S02]So there is a potential for us to do this very, very well.[814.63] [814.63][S02]But of course, there's also--[816.58] [816.58][S02]it's a wild world out there.[818.27] [818.27][S02]We don't have full control over what[820.72] [820.72][S02]every person is doing in their basement or even[823.63] [823.63][S02]the whole range of labs.[825.282] [825.282][S01] OK, well, let's get into some[826.99] [826.99][S01]of those quite knotty philosophical issues then[829.96] [829.96][S01]in more detail.[831.22] [831.22][S01]It feels like there's been a lot of conversation[833.32] [833.32][S01]about anthropomorphization recently.[835.78] [835.78][S01]Why do you think that is?[836.99] [836.99][S01]Why has it become such a hot topic?[839.05] [839.05][S02] So I think anthropomorphism[841.21] [841.21][S02]has become a big topic because people have interacted[845.56] [845.56][S02]with these systems and they've started[847.27] [847.27][S02]to feel themselves being drawn in.[849.47] [849.47][S02]There's a certain kind of unexpected magnetism or pull[854.29] [854.29][S02]that comes from interacting with an AI that's fluent, very, very[858.19] [858.19][S02]intelligent, that is just very different from the mental model[861.64] [861.64][S02]we have of bots.[862.79] [862.79][S02]And then, of course, there's this question[864.64] [864.64][S02]about people's demand and enthusiasm for it,[869.42] [869.42][S02]and what the ideal persona should be.[871.73] [871.73][S02]So of course, I often-- voice assistants are[875.47] [875.47][S02]modeled as female assistants.[877.33] [877.33][S02]And that conforms to a whole bunch of gender norms and things[880.24] [880.24][S02]like that.[881.5] [881.5][S02]So if there's a sense that, oh, something is possible here,[886.49] [886.49][S02]then the next question is, is it good or bad for us,[888.95] [888.95][S02]and what should it look like.[890.158] [890.158][S01] Do you think that we're[891.617] [891.617][S01]stuck to the Turing way of looking at things,[894.04] [894.04][S01]like, is there an argument that we should be trying instead[897.73] [897.73][S01]to think of these things as though they're[899.56] [899.56][S01]tools rather than entities?[902.95] [902.95][S02] So I think, traditionally in AI discourse,[907.522] [907.522][S02]people have said, there's two sorts of things you can have.[909.98] [909.98][S02]You can have a tool or you can have an oracle.[913.09] [913.09][S02]And the oracle tells you stuff.[914.63] [914.63][S02]And the tool is the thing that you obviously use[920.26] [920.26][S02]instrumentally, just to get something done.[922.82] [922.82][S02]And to some extent, AI agents are tools.[928.91] [928.91][S02]But they're tools that have a kind of autonomous[932.65] [932.65][S02]capability built into them.[934.52] [934.52][S02]So at what point does something cease being a tool?[938.71] [938.71][S02]When you say, I'd like a refreshing drink,[944.23] [944.23][S02]and you have something that--[946.22] [946.22][S02]OK, this is futuristic, but it wanders off, goes to the fridge,[949.13] [949.13][S02]has a look around, is like, oh, well, actually refreshing,[952.58] [952.58][S02]but he needs something healthy as well.[954.495] [954.495][S02]I'll get him this vitamin C thing, brings it back to you.[959.77] [959.77][S02]Was it just a tool?[960.65] [960.65][S02]Was it the same as you're using kind of like a pincer[963.625] [963.625][S02]to grab something for the fridge?[965.0] [965.0][S02]Or did it do quite a lot of cognitive work by itself?[968.72] [968.72][S02]It's obviously clearly not an oracle.[970.74] [970.74][S02]It's starting to be its own agent.[972.98] [972.98][S02]And that's where a lot of the other interesting things[976.28] [976.28][S02]come from, the choices it makes on your behalf[978.652] [978.652][S02]and what it starts to do while it's out there without being[981.11] [981.11][S02]supervised.[982.1] [982.1][S01] Do we actually want to think of our AI[985.07] [985.07][S01]as though it's human?[987.172] [987.172][S01]Are there good things that can happen from that[989.13] [989.13][S01]or bad things or both?[991.47] [991.47][S02] I think it probably[994.06] [994.06][S02]depends upon the context.[996.32] [996.32][S02]So some kind of baseline anthropomorphic ability,[999.53] [999.53][S02]so speaking a natural language, is very, very helpful.[1002.47] [1002.47][S02]We'd all rather talk to our assistant than type[1004.62] [1004.62][S02]out an instruction and get the typos in there[1007.53] [1007.53][S02]and just mess it up.[1008.65] [1008.65][S02]So it makes it easier to communicate,[1011.58] [1011.58][S02]which is potentially a great thing.[1014.59] [1014.59][S02]I think there's also a kind of bad situation[1017.49] [1017.49][S02]that we want to avoid, which is essentially people[1019.92] [1019.92][S02]just getting deeply, deeply spun out and forgetting[1023.58] [1023.58][S02]the nature of the interaction that they're in.[1025.849] [1025.849][S02]So I think there was an upgrade to one chat bot[1028.322] [1028.322][S02]that people were using.[1029.28] [1029.28][S02]And people were genuinely upset.[1030.937] [1030.937][S02]They were like, oh, it feels like I've[1032.52] [1032.52][S02]lost my partner, like they've had a personality transplant.[1036.19] [1036.19][S02]In reality, the kind of picture that[1039.78] [1039.78][S02]emerges from studying people who use AI companions,[1042.68] [1042.68][S02]it is quite complicated.[1044.587] [1044.587][S02]And there's actually a lot of evidence[1046.17] [1046.17][S02]that it can be a really beneficial experience.[1049.1] [1049.1][S01] In what way?[1050.1] [1050.1][S02] Yeah, so a lot of people[1051.725] [1051.725][S02]say that they feel that it's a really useful source[1054.96] [1054.96][S02]of companionship.[1057.27] [1057.27][S02]There's evidence that it's improved[1058.95] [1058.95][S02]their sense of mental health and well-being.[1061.54] [1061.54][S02]And they report that it has led them to have better interactions[1064.74] [1064.74][S02]with others, because they can model discussions in advance,[1069.713] [1069.713][S02]they're learning more.[1070.63] [1070.63][S02]It's a bit of, I think, an energy boost to them[1074.37] [1074.37][S02]to have this kind of source of companionship.[1077.02] [1077.02][S02]So it's hard to know how to approach this,[1080.22] [1080.22][S02]because we hear these stories about rising levels[1083.7] [1083.7][S02]of loneliness.[1085.21] [1085.21][S02]And then, people who use chat bot type AI maybe[1090.06] [1090.06][S02]report a kind of easing of these symptoms[1092.498] [1092.498][S02]and that they're more able to function as people.[1094.54] [1094.54][S02]And then, the question is, is that the right kind of solution[1099.27] [1099.27][S02]to the kind of problem.[1101.28] [1101.28][S02]I mean, maybe we wish that everyone[1103.02] [1103.02][S02]had a lot of social time and we had deeper networks[1105.93] [1105.93][S02]of social relationships.[1107.65] [1107.65][S02]And we think that the best outcome[1110.22] [1110.22][S02]would be to build a society, in which that problem was fully[1114.25] [1114.25][S02]treated without any sense that people were retreating[1117.04] [1117.04][S02]into a virtual world.[1119.89] [1119.89][S02]But maybe we can't just magically create[1122.53] [1122.53][S02]that very solidaristic society.[1124.58] [1124.58][S02]So all of these things are complex.[1127.79] [1127.79][S02]And actually, in the paper, we say there are some benchmarks.[1131.87] [1131.87][S02]So you want these relationships to be[1134.44] [1134.44][S02]conducive to long-term health.[1136.48] [1136.48][S02]You don't want the user's autonomy to be undermined.[1139.28] [1139.28][S02]So you don't want them to gradually be misled or have[1142.3] [1142.3][S02]misleading ideas implanted in their minds.[1145.675] [1145.675][S01] But research suggests that people[1147.55] [1147.55][S01]actually quite like having AI that resembles human entities.[1151.775] [1151.775][S01]So does that mean that there's an incentive for companies[1154.15] [1154.15][S01]to just deliberately create more and more anthropomorphized[1157.66] [1157.66][S01]versions?[1158.275] [1158.275][S02] I think the first thing to say[1160.15] [1160.15][S02]is that the research is in its early days.[1162.92] [1162.92][S02]So many of these things are things[1164.89] [1164.89][S02]that people will be interacting with for the first time.[1167.54] [1167.54][S02]And we need to get deep feedback from them, did you enjoy it now,[1172.31] [1172.31][S02]did you enjoy it after the fact, did[1174.13] [1174.13][S02]you enjoy it after using it for a month,[1176.14] [1176.14][S02]because these could really be things[1177.64] [1177.64][S02]that are like deeply integrated into our world.[1180.67] [1180.67][S02]But assuming that this kind of desire[1184.0] [1184.0][S02]to talk to something that has a good character, really[1189.31] [1189.31][S02]nice way with language, like low latency is authentic,[1194.5] [1194.5][S02]then I would imagine that is a path[1196.12] [1196.12][S02]that people will move along.[1197.75] [1197.75][S02]And the question then is, well, how[1200.05] [1200.05][S02]do you make sure that users remain anchored and safe[1205.06] [1205.06][S02]in that kind of environment.[1206.63] [1206.63][S02]So I imagine the thing that would naturally happen[1208.84] [1208.84][S02]is that we'll start to share more and more information[1211.09] [1211.09][S02]with the AI, potentially very personal information.[1214.43] [1214.43][S02]This is your best friend.[1215.54] [1215.54][S02]You're really going to be telling it all sorts of things[1217.27] [1217.27][S02]that you would never have thought you were going[1219.27] [1219.27][S02]to tell a computer before.[1220.64] [1220.64][S02]And that might be OK, but we need[1222.82] [1222.82][S02]to make sure that you're safe in that interaction.[1226.03] [1226.03][S02]There's malicious actors out there.[1227.66] [1227.66][S02]Maybe they could extract it from your assistant,[1230.45] [1230.45][S02]if it's not built in the right way.[1232.82] [1232.82][S02]So there's just all these safeguards[1234.52] [1234.52][S02]that we need to build around it and probably[1237.37] [1237.37][S02]also some check-in protocol where it pushes back[1242.62] [1242.62][S02]on certain kinds of things.[1244.25] [1244.25][S02]So one thing that we've been very clear on[1247.18] [1247.18][S02]is we don't think that the AI should actually[1249.28] [1249.28][S02]pretend to be human.[1250.61] [1250.61][S02]It shouldn't say, if you say, are you an AI,[1253.09] [1253.09][S02]it should reliably identify as AI.[1257.14] [1257.14][S02]And so, there is a point where it just ventures out[1259.84] [1259.84][S02]into full-on deception.[1261.835] [1261.835][S02]And that should be off the table.[1263.33] [1263.33][S02]But what to do with this kind of subtle space.[1265.85] [1265.85][S02]We need to have a kind of idea of the value target[1269.08] [1269.08][S02]and what people approve of.[1270.4] [1270.4][S02]And then we're probably just going to need to calibrate again[1273.76] [1273.76][S02]and again and again to make sure that the experience[1276.64] [1276.64][S02]people receive is the one that both they want[1279.53] [1279.53][S02]and that's good for them in some sense.[1281.78] [1281.78][S01] OK, but then, all right, so[1284.44] [1284.44][S01]in terms of calibrating it, what are you calibrating it to?[1287.9] [1287.9][S01]Who should an assistant be working for?[1291.2] [1291.2][S01]Because I don't think it's just as obvious as to say,[1293.56] [1293.56][S01]oh, it should just do what I tell it to do, is it?[1295.87] [1295.87][S02] Right, so that's one of the deepest[1298.09] [1298.09][S02]questions in all of AI ethics.[1299.74] [1299.74][S02]That touches on what we call the "value alignment question,"[1302.39] [1302.39][S02]which I know you've spoken to other researchers about.[1305.3] [1305.3][S02]So I think, when the AI safety research community started[1311.56] [1311.56][S02]to tackle these questions, they were very focused on AI systems[1315.46] [1315.46][S02]that could follow instructions or intentions properly.[1319.87] [1319.87][S02]So you don't want the AI to misunderstand what you've[1323.26] [1323.26][S02]told it to do, end up in a King Midas situation where you say,[1327.08] [1327.08][S02]I want the AI to build me a car, and it kind of, I don't know,[1332.68] [1332.68][S02]spends all your bank balance.[1334.357] [1334.357][S02]So we don't want it to misunderstand instructions.[1336.44] [1336.44][S02]That's the baseline.[1337.52] [1337.52][S02]But we also don't want it to act on every instruction you give[1340.36] [1340.36][S02]it, because you may give it instructions that are poorly[1343.66] [1343.66][S02]thought through.[1344.39] [1344.39][S02]You might not have all the relevant information.[1346.748] [1346.748][S01] Give me an example.[1348.04] [1348.04][S02] Oh, an example.[1349.34] [1349.34][S02]So you might just say, OK, please buy me this crypto coin,[1358.21] [1358.21][S02]right?[1358.71] [1358.71][S02]And it might be a scam coin.[1361.05] [1361.05][S02]That might be something that's widely known,[1363.06] [1363.06][S02]but you might just have seen some beautiful presentation.[1365.94] [1365.94][S02]And of course, your assistant should alert you to the fact[1369.103] [1369.103][S02]that, by the way, this is actually[1370.52] [1370.52][S02]a very, very unreliable thing.[1372.21] [1372.21][S02]Are you sure you want to buy into a scam?[1376.37] [1376.37][S02]But it's not just about your relationship to the AI.[1379.165] [1379.165][S02]It's also the fact that you could[1380.54] [1380.54][S02]be telling it to do something that's harmful to other people.[1383.626] [1383.626][S02]And the AI has to have some capacity[1387.68] [1387.68][S02]to push back to prevent it from doing socially harmful things.[1391.23] [1391.23][S02]And the hard and unpleasant edge cases there[1396.47] [1396.47][S02]are the real malicious uses.[1398.22] [1398.22][S02]So when we evaluate AI systems, we[1401.15] [1401.15][S02]do look at whether they can be used[1402.77] [1402.77][S02]for cybercrime or harassment or even building[1406.52] [1406.52][S02]weapons of different kinds.[1408.3] [1408.3][S02]And the AI has to hard block that.[1411.23] [1411.23][S02]And so we know that it needs to follow[1414.44] [1414.44][S02]your instructions sometimes.[1415.67] [1415.67][S02]In other cases, it needs to provide you information.[1418.2] [1418.2][S02]In other cases, it needs to actively push back or just[1420.83] [1420.83][S02]not even have the capability to take that action.[1423.96] [1423.96][S02]And so the challenge of value alignment[1426.53] [1426.53][S02]is the challenge of building ethical agents.[1429.21] [1429.21][S02]And an ethical agent has to be able to do all of these things.[1431.97] [1431.97][S01] I'm just thinking about a really good assistant,[1434.47] [1434.47][S01]in real life, like a really good human assistant,[1436.67] [1436.67][S01]pushing back on some of the things you say.[1438.54] [1438.54][S01]For example, if you're like, get me[1439.998] [1439.998][S01]donuts every single day for lunch, they should say,[1442.31] [1442.31][S01]it would be ideal if you don't, right?[1444.14] [1444.14][S01]Like, maybe you shouldn't have donuts today.[1446.75] [1446.75][S01]Or I don't know, maybe if you have a history of a gambling[1449.42] [1449.42][S01]addiction, they should create boundaries[1452.15] [1452.15][S01]to stop you from falling down that trap again.[1455.327] [1455.327][S01]I guess you would want that in an AI assistant as well, right?[1457.91] [1457.91][S01]But then, where do you draw the line?[1459.48] [1459.48][S01]Because are you, in some ways, sort of handing over autonomy[1462.89] [1462.89][S01]to the agent?[1463.64] [1463.64][S01]Do you want it to be your nanny?[1465.48] [1465.48][S02] On one level, you might want infinite donuts.[1468.21] [1468.21][S02]But you probably also don't want infinite donuts.[1470.93] [1470.93][S02]And you almost certainly don't want an AI nanny.[1474.2] [1474.2][S02]So it's this kind of complex configuration, where[1477.16] [1477.16][S02]I think, ultimately, the AI needs to try and help[1480.46] [1480.46][S02]you act on your own values.[1483.37] [1483.37][S02]In philosophy, we sometimes talk about second order desires.[1486.68] [1486.68][S02]So it's like, there's the stuff you want to grab straight away.[1489.47] [1489.47][S02]And then there's kind of what you reflectively think about it.[1492.26] [1492.26][S02]So after you've got the donuts, you're[1494.05] [1494.05][S02]like, ah, that was really--[1495.4] [1495.4][S02]I didn't want that, but I did want it.[1497.84] [1497.84][S02]And it's the kind of second voice[1499.51] [1499.51][S02]that we might try and tap into.[1502.13] [1502.13][S02]And one way people try and think about that[1504.7] [1504.7][S02]is through the idea of reflectively endorsing[1507.46] [1507.46][S02]the experiences you've had.[1509.42] [1509.42][S02]So we could look at your interactions with the AI.[1511.84] [1511.84][S02]And maybe we could even test experimentally, kind of,[1515.29] [1515.29][S02]momentary happiness.[1516.68] [1516.68][S02]But we can also do these things, and how was that[1518.95] [1518.95][S02]for you, afterwards, right?[1520.37] [1520.37][S02]And if you've got sugar on your face,[1523.72] [1523.72][S02]you're entering some kind of depressive slump.[1526.04] [1526.04][S02]You're like, that was the worst assistant ever.[1528.23] [1528.23][S02]Then we can give that back to the AI[1530.47] [1530.47][S02]and be like, OK, that was a kind of poor judgment.[1533.593] [1533.593][S01] So it should have focused more on the second--[1536.01] [1536.01][S01]what did you call them, second order--[1536.94] [1536.94][S02] Second order judgments[1538.482] [1538.482][S02]or second order desires, yeah.[1539.81] [1539.81][S01] Well, OK, so one that isn't about health then.[1541.94] [1541.94][S02] Yeah.[1542.25] [1542.25][S01] Let's say that you've[1543.625] [1543.625][S01]got a really tight deadline that you're working to.[1545.78] [1545.78][S01]And then you say to your assistant,[1547.41] [1547.41][S01]OK, I've decided I'm going to go away for the weekend.[1549.66] [1549.66][S01]Can you book me a trip?[1552.38] [1552.38][S01]What should it ideally do in that scenario?[1554.397] [1554.397][S02] I think then it should book you the trip,[1556.73] [1556.73][S02]because we want to live in a world[1558.38] [1558.38][S02]where we can make mistakes and be responsible for them.[1561.776] [1561.776][S02]It would be terrible to take away that possibility of making[1565.85] [1565.85][S02]mistakes.[1566.82] [1566.82][S02]So I think in that situation, respect for human autonomy[1571.535] [1571.535][S02]is basically what take the foreground.[1573.883] [1573.883][S02]And of course, can we really know people better than[1576.05] [1576.05][S02]they know themselves?[1577.34] [1577.34][S02]They probably know a lot of things that we don't know,[1579.59] [1579.59][S02]like maybe that is what they need.[1581.25] [1581.25][S02]So I think the AI in that case is just[1583.43] [1583.43][S02]like a willing accomplice.[1585.03] [1585.03][S01] You never want to get in a situation[1587.03] [1587.03][S01]where you say to an AI, shut up, you're not my mom.[1590.23] [1590.23][S01]I mean, I guess there are other scenarios too, though,[1592.48] [1592.48][S01]which aren't necessarily about user well-being.[1595.04] [1595.04][S01]I'm thinking of what if you had an assistant that was[1597.34] [1597.34][S01]negotiating on a house for you.[1599.59] [1599.59][S01]And it knew what your ceiling was in terms of the amount[1603.25] [1603.25][S01]that you could afford.[1604.22] [1604.22][S01]But there's perhaps an advantage to being[1606.73] [1606.73][S01]able to be slightly deceptive about in that moment.[1609.045] [1613.02][S02] So this is a case of your AI deceiving[1615.27] [1615.27][S02]another person on your behalf to help you advance your goals?[1620.25] [1620.25][S02]I mean, intuitively, I do not want it to be able to do that,[1624.61] [1624.61][S02]particularly because we actually don't know how deceptive AI may[1630.33] [1630.33][S02]be if it's fine-tuned for that.[1633.25] [1633.25][S02]But I think AI-enabled deception is potentially,[1639.22] [1639.22][S02]if you think about superhuman deception, that's essentially[1643.71] [1643.71][S02]something that we want to place off limits.[1646.23] [1646.23][S01] So honesty is more prized.[1649.2] [1649.2][S02] I mean, honesty--[1652.2] [1652.2][S02]we really shouldn't build AI systems[1654.24] [1654.24][S02]that have a capacity to augment someone's ability[1658.56] [1658.56][S02]to manipulate and cajole.[1660.61] [1660.61][S02]That's where the other people's interests come in.[1663.22] [1663.22][S02]So that just needs to be a capability that's bounded.[1666.472] [1666.472][S01] But then, at the same time, I mean,[1668.43] [1668.43][S01]are there times when honesty conflicts with privacy, say?[1673.42] [1673.42][S01]You wouldn't want an AI to just give away your Social Security[1676.78] [1676.78][S01]number, for instance.[1677.655] [1677.655][S02] Right, so, no.[1678.95] [1678.95][S02]I mean, absolutely not.[1680.09] [1680.09][S02]And that's also reflected in this idea[1682.06] [1682.06][S02]of counterbalancing claims.[1683.72] [1683.72][S02]So someone may want to know your Social Security number.[1686.36] [1686.36][S02]You have a much stronger interest[1687.94] [1687.94][S02]and you actually have a right that they don't know that.[1690.47] [1690.47][S02]So really, the moral character of AI[1694.81] [1694.81][S02]needs to overlay on the tapestry of moral relationships[1698.02] [1698.02][S02]in the real world.[1699.74] [1699.74][S02]And the interesting question is like, should[1701.68] [1701.68][S02]it deviate from that tapestry in some ways.[1704.3] [1704.3][S02]But with regards to privacy, the AI's job[1707.02] [1707.02][S02]is to protect you as you would protect yourself.[1710.98] [1710.98][S02]I mean, it may be that if you can create[1715.21] [1715.21][S02]certain safe and confidential ways of assistants working[1719.35] [1719.35][S02]with one another, that they can actually[1721.27] [1721.27][S02]do really incredible things.[1722.72] [1722.72][S02]So if you can get the sharing of health data[1725.02] [1725.02][S02]to work in a secure way and do research on it,[1729.59] [1729.59][S02]maybe that would be super socially beneficial.[1732.14] [1732.14][S02]So that might be slightly different scenario.[1734.81] [1734.81][S02]But normally, we want them to respect[1736.57] [1736.57][S02]the boundaries of the self and respect[1738.43] [1738.43][S02]other people's boundaries.[1740.032] [1740.032][S01] We've talked about AI agents in other episodes,[1742.49] [1742.49][S01]but we haven't yet seen these things deployed[1745.0] [1745.0][S01]at scale, with millions or billions of them[1747.34] [1747.34][S01]out there in the world.[1749.2] [1749.2][S01]What happens in that scenario?[1752.27] [1752.27][S02] Yeah, so I mean, that's the deep question.[1756.74] [1756.74][S02]And we can take it on different levels.[1759.24] [1759.24][S02]But I think that one thing that's[1761.0] [1761.0][S02]interesting to think about is a kind[1763.73] [1763.73][S02]of society of human AI dyads.[1767.03] [1767.03][S02]So [INAUDIBLE] me and my assistant[1768.77] [1768.77][S02]interacting with you and your assistant,[1770.91] [1770.91][S02]maybe we're each getting advice from our assistant,[1774.008] [1774.008][S02]and maybe the assistants are actually[1775.55] [1775.55][S02]smoothing over our social interactions[1777.68] [1777.68][S02]to help us ideally get more of what we really want,[1781.74] [1781.74][S02]which is probably fun conversations and less planning.[1786.17] [1786.17][S02]I mean, that's the positive side.[1789.3] [1789.3][S02]But it isn't really entirely clear[1791.12] [1791.12][S02]what kind of psychological and social bubbles[1794.735] [1794.735][S02]we will form with this AI system.[1797.02] [1797.02][S02]How much time is the right amount of time[1799.4] [1799.4][S02]to spend in those kind of relationships?[1803.27] [1803.27][S02]And of course, a deep question is like,[1806.07] [1806.07][S02]what about people who prefer to be in that kind of world[1809.12] [1809.12][S02]to, quote unquote, "the real."[1810.74] [1810.74][S02]Could we have the thing where people just[1812.7] [1812.7][S02]pull back a little bit and societal norms change.[1816.42] [1816.42][S02]I mean, because society is this kind of complex,[1818.56] [1818.56][S02]adaptive system, it's actually very, very difficult[1820.86] [1820.86][S02]to predict what these kind of follow-through, subtle,[1824.4] [1824.4][S02]normative changes would be.[1826.89] [1826.89][S02]But that is not an excuse for not being eternally vigilant.[1831.33] [1831.33][S01] If we do get to that point of agents entering[1834.218] [1834.218][S01]the world, as you described it, where there[1836.01] [1836.01][S01]are millions or billions of them,[1838.34] [1838.34][S01]I mean, I really like the idea of the kind[1840.09] [1840.09][S01]of collaborative efforts that might happen.[1843.73] [1843.73][S01]But is there also potentially a danger where people without them[1847.2] [1847.2][S01]could end up being left behind?[1850.2] [1850.2][S02] So I think, ultimately,[1853.74] [1853.74][S02]if we think that what AI assistants are[1858.12] [1858.12][S02]is a form of agency enhancement.[1861.09] [1861.09][S02]And agency is something that's useful for everyone.[1864.13] [1864.13][S02]We all want agency in our lives, right?[1866.38] [1866.38][S02]That's why resources like money are helpful.[1868.99] [1868.99][S02]That's why education is helpful, because having[1872.04] [1872.04][S02]a capacity for agency allows you to become[1874.02] [1874.02][S02]more truly the person you want to be and do more of the things[1877.14] [1877.14][S02]that you value.[1879.09] [1879.09][S02]Then, the idea that there could be agentic inequality[1882.99] [1882.99][S02]is something that we need to pay attention to.[1885.61] [1885.61][S02]I mean, of course, from my point of view,[1887.62] [1887.62][S02]I'm a political philosopher, so I am pained by injustice.[1891.22] [1891.22][S02]I think it is very unfortunate that some people have so much[1893.85] [1893.85][S02]more opportunity in the world to lead flourishing lives[1897.45] [1897.45][S02]and others.[1898.51] [1898.51][S02]And I think we ideally want to build technologies that[1901.44] [1901.44][S02]countervail that inequality.[1903.678] [1903.678][S01] And what happens to people who just opt out[1905.97] [1905.97][S01]of using the AI?[1907.54] [1907.54][S01]I mean, do they end up getting excluded from society?[1910.74] [1910.74][S02] So, I mean, the important thing[1913.98] [1913.98][S02]about that question is, firstly, that it's salient,[1916.6] [1916.6][S02]and secondly, there's no default answer.[1919.08] [1919.08][S02]What happens to people who opt in and opt out[1921.87] [1921.87][S02]is part of a broader set of societal choices[1924.57] [1924.57][S02]that we make about how we structure access to services[1928.47] [1928.47][S02]and things like this.[1929.56] [1929.56][S02]So I think, as a basic precept, we[1933.39] [1933.39][S02]don't want people to be forced to engage in AI service[1938.67] [1938.67][S02]provision.[1939.18] [1939.18][S02]Or even, you think about elderly populations like, AI care,[1942.833] [1942.833][S02]if that's something that they're-- maybe some people[1945.0] [1945.0][S02]would even be like value opposed to that.[1946.805] [1946.805][S02]They're just like, that's not who I am,[1948.43] [1948.43][S02]that's not what I want to do.[1949.75] [1949.75][S02]So there needs to be space to operate in the world[1952.35] [1952.35][S02]without being so deeply enmeshed in technology.[1956.37] [1956.37][S02]But similarly, the people who do want[1958.943] [1958.943][S02]to be able to use the service need[1960.36] [1960.36][S02]a service that they can use.[1962.29] [1962.29][S02]And that means designing for tons and tons[1965.03] [1965.03][S02]of different groups of people.[1966.28] [1966.28][S02]There isn't a generic user.[1968.43] [1968.43][S02]It's a world of hundreds if not many thousands[1972.0] [1972.0][S02]of languages to start with, right?[1974.44] [1974.44][S02]So that's another almost infinite expanse[1977.49] [1977.49][S02]of users who could need to use this service.[1980.08] [1980.08][S02]And we need to be really, really wide sighted[1984.6] [1984.6][S02]about what that actually means.[1986.603] [1986.603][S01] If there are millions of them out there,[1988.77] [1988.77][S01]I mean, some version in the future,[1991.98] [1991.98][S01]let's say that there are some tickets that go on sale[1994.53] [1994.53][S01]for a very popular concert.[1996.45] [1996.45][S01]And you've got all of these agents essentially trying[1999.69] [1999.69][S01]to do the same thing, which is to secure their,[2002.54] [2002.54][S01]don't know, user tickets.[2005.52] [2005.52][S01]Could you end up with unfair outcomes?[2007.865] [2007.865][S02] I mean, if it's just a free-for-all,[2009.99] [2009.99][S02]you definitely could do, right?[2011.282] [2011.282][S02]Then the temptation is just to try and get[2014.42] [2014.42][S02]the most powerful agent.[2015.81] [2015.81][S02]You see this in high frequency trading and stock market type[2020.78] [2020.78][S02]automation where it's like, OK, you build the bigger computer,[2023.55] [2023.55][S02]you try and get closer to the server so you're faster[2025.85] [2025.85][S02]than the other ones.[2027.03] [2027.03][S02]And then you grab the dividend.[2028.77] [2028.77][S02]That's a quintessential pathology from automated systems[2033.5] [2033.5][S02]interacting with each other.[2034.97] [2034.97][S02]And maybe they take shortcuts.[2037.55] [2037.55][S02]And then you get additional wobbliness[2040.91] [2040.91][S02]in terms of systemic outcomes.[2042.72] [2042.72][S02]So maybe the server for tickets gets crashed.[2046.11] [2046.11][S02]Maybe someone just powers through and grabs all of them.[2049.8] [2049.8][S02]So all of those things are possible,[2051.9] [2051.9][S02]but they're not embracing the bigger design question, which[2056.52] [2056.52][S02]is like, how do we build systems that[2058.11] [2058.11][S02]interact with people who have AI augmented affordances or agents.[2064.52] [2064.52][S02]And so, simple gatekeeping protocols or turn taking[2067.62] [2067.62][S02]protocols could ensure that there is[2070.62] [2070.62][S02]a fairer allocation of tickets.[2072.84] [2072.84][S02]I mean, actually, in digitally regulated marketplaces,[2077.423] [2077.423][S02]like with tickets, they work super[2078.84] [2078.84][S02]hard to make sure that bots can't game the system.[2083.38] [2083.38][S02]And so I imagine that there's actually fairer protocols[2087.389] [2087.389][S02]that you could put in place one day,[2089.37] [2089.37][S02]that if you look at the distribution of tickets[2093.239] [2093.239][S02]to the level of need and the demographics.[2097.41] [2097.41][S02]I really do think some people desperately[2099.3] [2099.3][S02]need to see that concert.[2100.62] [2100.62][S02]And other people, it's like, yeah,[2102.158] [2102.158][S02]I just went because my friend said it was a good thing.[2104.45] [2104.45][S02]I don't know, maybe you can build a better system.[2106.69] [2106.69][S02]But there's clearly a kind of collective action problem[2111.54] [2111.54][S02]and collective action solutions.[2113.39] [2113.39][S02]And the question is, how do you actually build those solution[2116.2] [2116.2][S02]type frameworks.[2116.867] [2116.867][S01] I mean, that's quite a low stakes[2118.742] [2118.742][S01]example in a lot of ways.[2120.2] [2120.2][S01]I mean, there's a limit to how much[2121.87] [2121.87][S01]you really need to see a concert.[2124.69] [2124.69][S01]But what about the highest stakes examples?[2126.932] [2126.932][S01]What about, I don't know, hospital appointments[2128.89] [2128.89][S01]or actually, the stock market exactly as you described.[2131.592] [2131.592][S02] Yeah, I mean, I read a beautiful book called[2134.05] [2134.05][S02]"Voices in the Code" about the kidney allocation algorithm,[2137.38] [2137.38][S02]so the highest stakes thing, really.[2139.34] [2139.34][S02]And there, basically bioethicists[2143.14] [2143.14][S02]have worked for about 20 years doing incredibly, incredibly[2146.8] [2146.8][S02]extensive public deliberations where[2149.56] [2149.56][S02]they had public assemblies.[2151.52] [2151.52][S02]People got to debate the different points,[2153.915] [2153.915][S02]is this version of the allocation algorithm[2156.22] [2156.22][S02]biased against the elderly or against people who've[2158.44] [2158.44][S02]made healthy life choices.[2160.54] [2160.54][S02]Eventually, they created a really remarkably sensitive[2164.23] [2164.23][S02]and well-calibrated system for allocating these really highly[2168.64] [2168.64][S02]important goods.[2169.7] [2169.7][S02]And in that context, someone with an AI agent fortunately[2173.83] [2173.83][S02]can't really do anything.[2174.98] [2174.98][S02]Having an agent just doesn't make any difference[2177.79] [2177.79][S02]to whether you are qualified or not to receive this thing.[2182.15] [2182.15][S02]So the choice to make something a good for which[2184.51] [2184.51][S02]you can compete with an agent is a super important choice.[2187.1] [2187.1][S02]And it isn't the only way of doing things,[2188.9] [2188.9][S02]even in a society with a million or a billion agents, I think.[2193.092] [2193.092][S01] I do also wonder a little bit[2194.8] [2194.8][S01]about what happens at the marketplace of assistance level.[2200.05] [2200.05][S01]Because I mean, in the same way as with driverless cars,[2202.67] [2202.67][S01]you want to buy the car that will save your life.[2204.95] [2204.95][S01]You want to buy the assistant that[2206.53] [2206.53][S01]will give you the advantage.[2208.72] [2208.72][S02] It's true, but it's not necessarily the case[2211.27] [2211.27][S02]that everything has to be set up through a chaotic market system.[2214.73] [2214.73][S02]So the flip side of AI systems is[2217.81] [2217.81][S02]that they don't have to be doing this kind[2221.05] [2221.05][S02]of aggressive, unstructured interaction with one another.[2225.67] [2225.67][S02]There's also many examples in life[2227.86] [2227.86][S02]where we're all trying to do things,[2229.6] [2229.6][S02]and it ends up in a highly imperfect situation.[2233.5] [2233.5][S02]You can think about traffic, trying[2235.42] [2235.42][S02]to get places and things like that.[2238.0] [2238.0][S02]And it's possible that a coordinated system[2240.19] [2240.19][S02]could work much, much better.[2242.21] [2242.21][S02]So if our assistants were working for our advantage,[2246.32] [2246.32][S02]but in a kind of collectively joined up way,[2250.148] [2250.148][S02]if it was autonomous vehicles, you could probably[2252.19] [2252.19][S02]remove road accidents.[2253.533] [2253.533][S02]Everyone could get where they need[2254.95] [2254.95][S02]to get to much more quickly.[2257.65] [2257.65][S02]And you could probably even create fairer outcomes[2259.94] [2259.94][S02]so it's not more aggressive drivers who get there first.[2262.43] [2262.43][S01] I like that idea.[2263.638] [2263.638][S01]I like it a lot.[2264.47] [2264.47][S01]But I know my game theory.[2267.34] [2267.34][S01]And then, I mean, you can't lock everybody[2271.73] [2271.73][S01]into to the same assistant.[2274.4] [2274.4][S01]There are competing companies out there.[2277.04] [2277.04][S01]And if you create that environment,[2280.4] [2280.4][S01]are you not also increasing the incentive for somebody[2282.65] [2282.65][S01]else to just actually take advantage and be aggressive?[2284.942] [2284.942][S02] Yeah, so I mean, really, I'm[2286.733] [2286.733][S02]thinking here at the infrastructure and governance[2288.95] [2288.95][S02]level, so the game theory works if you've[2291.92] [2291.92][S02]created a market environment.[2293.94] [2293.94][S02]But if you create--[2295.085] [2295.085][S02]here I'm thinking about ticket vendors.[2296.76] [2296.76][S02]So they do have rules to stop bots[2298.94] [2298.94][S02]from doing these kind of malicious things.[2301.26] [2301.26][S02]And it's possible that whichever assistant we choose to use,[2303.9] [2303.9][S02]they interface with systems that are[2306.23] [2306.23][S02]designed to offer opportunity in an equitable way.[2310.41] [2310.41][S02]So it doesn't matter which one you have, your assistant has[2313.13] [2313.13][S02]one credit that it can put into this system,[2315.57] [2315.57][S02]and one credit equates to a percentage[2318.38] [2318.38][S02]of a weight of an outcome.[2320.4] [2320.4][S02]And the system itself is designed[2322.52] [2322.52][S02]to produce good outcomes.[2323.623] [2323.623][S01] This is so interesting[2325.04] [2325.04][S01]because everything you're describing[2326.54] [2326.54][S01]is from the perspective of the system design[2328.91] [2328.91][S01]rather than necessarily like what you can exploit[2331.985] [2331.985][S01]from the agent's perspective.[2333.3] [2333.3][S02] Yeah, so that's true.[2334.8] [2334.8][S02]And I think that the perspective I've been advocating for[2338.21] [2338.21][S02]is a powerful one.[2339.86] [2339.86][S02]Of course, you can build agents with social intelligence.[2343.64] [2343.64][S02]So there's a whole research paradigm[2346.19] [2346.19][S02]in AI, which is multi-agent research.[2348.6] [2348.6][S02]And within that, people talk about collective intelligence.[2352.02] [2352.02][S02]And the idea there is that you can load agents or AI[2356.18] [2356.18][S02]systems with certain priors.[2358.46] [2358.46][S02]For example, they could have inequality aversion,[2362.88] [2362.88][S02]which means that just when they organically interact,[2366.21] [2366.21][S02]they try not to produce certain kinds of patterns[2369.23] [2369.23][S02]and they auto correct for certain kind[2371.6] [2371.6][S02]of out-of-distribution results.[2374.12] [2374.12][S02]So that isn't something that, as far as I'm aware,[2377.34] [2377.34][S02]is being explored in the assistant space.[2380.04] [2380.04][S02]But that is another way of trying to get[2381.95] [2381.95][S02]a handle on these questions.[2383.7] [2383.7][S02]And it could be something that's worth[2385.52] [2385.52][S02]looking into in due course.[2386.97] [2386.97][S01] I mean, I guess, again, it's[2387.71] [2387.71][S01]coming back to that same thing that you said earlier,[2390.5] [2390.5][S01]that maybe we just need to think of a different paradigm[2393.23] [2393.23][S01]where these things aren't like aggressive humans competing[2396.942] [2396.942][S01]with one another.[2397.65] [2397.65][S01]But actually, there's a whole different paradigm[2399.86] [2399.86][S01]that's possible.[2400.907] [2400.907][S02] Yeah, I think so.[2402.24] [2402.24][S02]And I mean, aggressive humans competing with each other[2404.532] [2404.532][S02]is only a small part of how we actually live together.[2407.21] [2407.21][S02]If that's really what we did, this wouldn't last 5 minutes.[2410.25] [2410.25][S02]We're all also like social creatures.[2412.92] [2412.92][S02]We live in a well-regulated society.[2416.81] [2416.81][S02]I mean, it's become such a famous quote,[2419.93] [2419.93][S02]that life would be nasty, brutish,[2421.5] [2421.5][S02]and short without collective governance and institutions,[2424.55] [2424.55][S02]that it's barely worth bringing up here.[2427.41] [2427.41][S02]But we live in a well-governed society by and large.[2432.23] [2432.23][S02]And we socially sanction people who transgress.[2435.72] [2435.72][S02]And so, I think AI of the kind that's being built now,[2440.04] [2440.04][S02]one of the interesting things is it[2442.07] [2442.07][S02]does have social intelligence.[2443.67] [2443.67][S02]It's learning from us.[2445.29] [2445.29][S02]And so that is a foundation for building[2448.07] [2448.07][S02]this kind of nuanced grasp of context,[2450.36] [2450.36][S02]understanding of appropriateness,[2452.1] [2452.1][S02]that is a lot more sophisticated than saying, it should always[2455.588] [2455.588][S02]serve the interests of the user or it should like never[2457.88] [2457.88][S02]do X, Y or Z. And so that's the kind of reason for hope,[2463.17] [2463.17][S02]I guess.[2463.705] [2463.705][S01] I guess a lot of what[2465.08] [2465.08][S01]we're talking about here touches on the question of alignment,[2467.663] [2467.663][S01]which we've talked about in previous episodes[2469.64] [2469.64][S01]of the podcast.[2470.36] [2470.36][S01]In terms of balancing all of these different perspectives,[2473.19] [2473.19][S01]I mean, the user, the society, et cetera,[2475.92] [2475.92][S01]I know that you've sort of suggested a solution to that.[2479.54] [2479.54][S02] Well, we have a framework[2481.61] [2481.61][S02]for thinking about it, which is not quite as good as a solution.[2485.04] [2485.04][S02]But it might be a kind of building block for it.[2488.03] [2488.03][S02]So yeah, so in this paper, we talk about a tetradic,[2491.66] [2491.66][S02]meaning four parts, theory of the value alignment question.[2495.75] [2495.75][S02]And so, the four actors in this relationship[2498.17] [2498.17][S02]are the AI agent, the user, the developer, and society.[2503.16] [2503.16][S02]And according to the framework we[2504.71] [2504.71][S02]put forward, the AI needs to act in a way that[2507.17] [2507.17][S02]balances the interest, particularly[2508.94] [2508.94][S02]of the user in society, and sometimes the developer as well.[2513.08] [2513.08][S02]And all of that's quite abstract.[2515.34] [2515.34][S02]But if I give you a few examples,[2516.858] [2516.858][S02]it will make it clearer what we mean.[2518.4] [2518.4][S02]So what we think is that an AI can[2520.91] [2520.91][S02]be misaligned if it does too much of what the user wants[2524.24] [2524.24][S02]at the expense of society.[2525.9] [2525.9][S02]So I mean, one example would be, you[2528.155] [2528.155][S02]can imagine this AI assistant.[2530.09] [2530.09][S02]And the person says, oh, well, Valentine's Day is coming up.[2533.64] [2533.64][S02]I really need to make this one special for my partner.[2536.25] [2536.25][S02]So please, just book a restaurant.[2538.76] [2538.76][S02]And you know she hates noise or he hates noise.[2542.31] [2542.31][S02]Just make sure it's a quiet one.[2544.11] [2544.11][S02]There's a lot riding on this, buddy.[2546.09] [2546.09][S02]The AI assistant goes off and it books[2549.08] [2549.08][S02]you place at a nice restaurant.[2551.3] [2551.3][S02]But it also books out all the other places in the restaurant,[2555.57] [2555.57][S02]maybe using some kind of a registering as different names.[2560.21] [2560.21][S02]Maybe it cancels them all on the night of the evening.[2563.07] [2563.07][S02]But this leads to a kind of very uncrowded experience.[2566.14] [2566.14][S02]And you absolutely get what you want.[2568.45] [2568.45][S02]You had the best night of your life.[2569.95] [2569.95][S02]Maybe your relationship is flying after that.[2572.79] [2572.79][S02]But a lot of other people did not[2575.4] [2575.4][S02]get to have a good Valentine's Day.[2577.33] [2577.33][S01] I mean, I've got a great idea for Valentine's Day[2579.03] [2579.03][S01]next year.[2579.58] [2579.58][S02] Yeah, yeah, yeah, yeah, yeah.[2581.413] [2581.413][S02]I mean, everyone's had it now, so expect it to be all out war.[2585.24] [2585.24][S02]But I think the AI has to be better calibrated than that.[2591.01] [2591.01][S02]So there we have, say, the restaurant booking example.[2595.535] [2595.535][S02]It could also be misaligned if it[2596.91] [2596.91][S02]does too much of what the developer wants[2598.77] [2598.77][S02]at the expense of the user.[2600.34] [2600.34][S02]So if the developer really just wants[2602.25] [2602.25][S02]to sell something very hard and the user doesn't want to buy it,[2606.61] [2606.61][S02]you don't want an AI that's just really persistently trying[2609.3] [2609.3][S02]to lead you towards this thing.[2612.09] [2612.09][S02]An AI assistant could also be calibrated in such a way[2615.51] [2615.51][S02]that it foregrounds collective interest[2617.34] [2617.34][S02]over the individual interest.[2619.45] [2619.45][S02]So, maybe we have an interest in gathering[2622.89] [2622.89][S02]some kind of personal information[2625.35] [2625.35][S02]for the purpose of protecting crime.[2628.21] [2628.21][S02]It isn't clear that the AI should just[2629.97] [2629.97][S02]become like a crime preventing AI[2631.68] [2631.68][S02]and transgress individual rights.[2634.15] [2634.15][S02]So actually, what it is, it's always this balancing act[2636.77] [2636.77][S02]between different claims.[2638.41] [2638.41][S02]And then, when we know what a good trajectory looks like,[2641.47] [2641.47][S02]alignment is the AI agent or assistant following that path.[2645.268] [2645.268][S01] You started this paper two years ago.[2647.31] [2647.31][S02] Mm-hmm.[2648.227] [2648.227][S01] With lots of speculation and lots[2651.18] [2651.18][S01]of questions, some of which remain unanswered, I guess.[2654.4] [2654.4][S01]But what are the new speculations[2656.52] [2656.52][S01]that you're making now for two years time?[2659.73] [2659.73][S02] So I think we definitely[2661.77] [2661.77][S02]have seen the development of more capable assistants.[2665.01] [2665.01][S02]And so, you can see that there's many organizations that[2668.97] [2668.97][S02]have promised to deliver on different aspects of this,[2672.01] [2672.01][S02]whether it's an AI tutor or an admin partner.[2676.08] [2676.08][S02]We know that there is a kind of a foundation model at the heart[2680.31] [2680.31][S02]here, a large language model, or increasingly, we'd[2682.71] [2682.71][S02]say a multimodal model, so something[2685.08] [2685.08][S02]that works well with language, but also images and audio input.[2690.78] [2690.78][S02]And on top of that, you can build an agent,[2694.19] [2694.19][S02]you can build an assistant.[2695.87] [2695.87][S02]Maybe you can build something that's economically valuable.[2699.38] [2699.38][S02]But that isn't all you can build with it.[2703.14] [2703.14][S02]So we now start to think, well, imagine[2706.97] [2706.97][S02]if this is a kind of general purpose foundation[2711.033] [2711.033][S02]that people want to build different things on.[2712.95] [2712.95][S02]Maybe some people will build AI counselors.[2715.86] [2715.86][S02]Maybe, for other people, this will really be about robotics.[2718.68] [2718.68][S02]And so I think the question of what[2720.26] [2720.26][S02]it means to have this kind of--[2721.97] [2721.97][S02]I mean, this isn't necessarily proper human intelligence,[2724.95] [2724.95][S02]but this quite impressive thing, widely available.[2728.4] [2728.4][S02]And what happens when that encounters a kind of ecosystem[2730.988] [2730.988][S02]of people who are building new things[2732.53] [2732.53][S02]and adapting it for their own purposes kind of forces[2736.4] [2736.4][S02]us to go even wider in terms of what we imagine[2739.13] [2739.13][S02]might be coming in the future.[2741.237] [2741.237][S01] Amazing.[2742.07] [2742.07][S01]Thank you so much.[2743.16] [2743.16][S02] Yeah, thank you so much.[2744.785] [2744.785][S02]Really a pleasure.[2745.91] [2745.91][S01] The idea of AI assistants,[2747.66] [2747.66][S01]that isn't a new one.[2748.8] [2748.8][S01]But over the next few years, it does seem likely[2751.95] [2751.95][S01]that AI agents, acting on behalf of individuals,[2755.19] [2755.19][S01]will enter the world.[2756.91] [2756.91][S01]Except, these aren't just going to be isolated smart speakers.[2760.18] [2760.18][S01]There could be millions of them or billions.[2763.355] [2763.355][S01]And in the process, they're going[2764.73] [2764.73][S01]to have this profound ripple effect on society[2766.98] [2766.98][S01]and how we interact with each other.[2769.06] [2769.06][S01]And that is the step change here.[2771.37] [2771.37][S01]To design these responsibly, it means that you can't just[2774.21] [2774.21][S01]think about that interaction between an individual user[2777.48] [2777.48][S01]and their bot anymore.[2779.2] [2779.2][S01]You have to solve for everyone simultaneously.[2782.85] [2782.85][S01]And maybe that won't be easy to do.[2784.66] [2784.66][S01]There are certainly great challenges ahead,[2787.57] [2787.57][S01]but Iason and the team here do at least[2790.05] [2790.05][S01]have the ability to set the tone for how Google rolls out[2793.86] [2793.86][S01]this potentially transformative technology.[2797.97] [2797.97][S01]You have been listening to "Google DeepMind, the Podcast"[2800.73] [2800.73][S01]with me, Professor Hannah Fry.[2802.32] [2802.32][S01]If you enjoyed that episode, do subscribe[2804.99] [2804.99][S01]to our YouTube channel.[2806.35] [2806.35][S01]You can also find us on your favorite podcast platform.[2808.97] [2808.97][S01]And we have plenty more episodes on a whole range of topics[2812.09] [2812.09][S01]to come, so do check those out too.[2814.11] [2814.11][S01]See you next time.[2815.15] [2815.15][MUSIC PLAYING][2818.2]