[0.0][MUSIC PLAYING][4.275] [6.65][S01] Welcome to "Google DeepMind-- the Podcast."[9.15] [9.15][S01]I'm Professor Hannah Fry.[10.68] [10.68][S01]Right, agents, they are here, or almost here,[15.21] [15.21][S01]and they are probably all anyone is going[18.41] [18.41][S01]to be talking about in 2025.[20.97] [20.97][S01]But they are definitely not new.[24.42] [24.42][S01]My guest on today's episode is someone[26.84] [26.84][S01]who last came on the podcast in 2019[30.17] [30.17][S01]to talk to me about the multi-agent system he[33.08] [33.08][S01]was working on.[34.11] [34.11][S01]It could beat the professional StarCraft players[36.71] [36.71][S01]at their own game, and eventually went on[38.81] [38.81][S01]to achieve Grandmaster status.[40.98] [40.98][S01]But how have agents evolved since then?[44.64] [44.64][S01]What can they do now?[46.02] [46.02][S01]How have the advances in language models[48.35] [48.35][S01]and multimodal AI change things?[51.06] [51.06][S01]And how do you possibly go about building something[55.37] [55.37][S01]that can make autonomous decisions on behalf of its user?[59.73] [59.73][S01]Now, I should tell you, if you want a primer on agents,[62.84] [62.84][S01]you can watch our episode with Frederic Besse[64.989] [64.989][S01]that we recorded over the summer.[67.07] [67.07][S01]But for now, Oriol Vinyals is Vice President[70.39] [70.39][S01]of Drastic Research and co-tech-lead of Gemini.[73.91] [73.91][S01]And it's fair to say we've got quite a lot to catch up on.[76.88] [76.88][S01]Oriol, welcome back to the podcast.[78.56] [78.56][S02] Hi, thank you for having me.[79.88] [79.88][S01] What is Drastic Research?[81.56] [81.56][S02] Well, I keep telling[83.2] [83.2][S02]my team they have to think drastic,[85.03] [85.03][S02]meaning don't just do the incremental stuff[88.12] [88.12][S02]that everyone is thinking about.[89.75] [89.75][S02]Try to drastically think what will happen in a few years time,[94.19] [94.19][S02]and then try to backport those ideas[96.97] [96.97][S02]and then execute today with that mindset in mind.[99.95] [99.95][S02]So that's what drastic means.[101.205] [101.205][S02]But yeah, it's a word that I use a lot.[102.83] [102.83][S01] I think when I last got to see you,[104.788] [104.788][S01]you had been working on an agent that[107.14] [107.14][S01]could use a keyboard and a mouse to do things draw pictures[111.37] [111.37][S01]in Paint or play StarCraft.[114.71] [114.71][S01]And, well, things have moved on quite a bit since then.[117.85] [117.85][S02] So those agents, at the time,[120.19] [120.19][S02]you took sort of a very generic set of principles,[123.61] [123.61][S02]very simple principles in the field of machine learning,[127.75] [127.75][S02]and you would basically specialize a model on one task.[134.29] [134.29][S02]And what we were doing at the time[136.62] [136.62][S02]is have a curriculum of tasks that were more and more and more[140.46] [140.46][S02]difficult, right?[141.25] [141.25][S02]So when we last spoke, for instance, in video games,[144.58] [144.58][S02]we were looking at StarCraft, which[146.28] [146.28][S02]is one of the most complex modern strategy games out there.[150.4] [150.4][S02]And of course, DeepMind is notorious[151.98] [151.98][S02]for having started the trend with Atari, which[154.38] [154.38][S02]is a fairly simple game of left, right, hit the paddle[157.17] [157.17][S02]and hit the ball and off you go.[159.85] [159.85][S02]So that is what the algorithms themselves, we[164.94] [164.94][S02]tried to push for them to be very general so we can keep[168.42] [168.42][S02]climbing this ladder of difficulty, curriculum of games,[172.05] [172.05][S02]and doing more and more complex things.[174.91] [174.91][S02]And right now what has happened is even[178.62] [178.62][S02]the models we train are broadly applicable to many more things[182.76] [182.76][S02]than the models we developed back then were.[185.92] [185.92][S02]So think about the process of creating this digital brain[189.54] [189.54][S02]hasn't changed that much.[191.11] [191.11][S02]But what that brain was able to do[193.38] [193.38][S02]was reasonably narrow, although very complex,[196.18] [196.18][S02]like playing StarCraft or playing Go.[198.27] [198.27][S02]Right now, these models can do quite a lot more broad[203.64] [203.64][S02]applications and, of course, talking to us, chatbots, et[207.39] [207.39][S02]cetera, et cetera.[208.39] [208.39][S01] So back then, reinforcement learning[210.57] [210.57][S01]was your main lever, I guess.[213.04] [213.04][S01]How different are things now?[214.457] [214.457][S02] Yeah, so algorithmically, actually,[216.54] [216.54][S02]the process of AlphaGo and actually AlphaStar, those two[221.49] [221.49][S02]had the same set of sequence of algorithms applied[224.88] [224.88][S02]to creating this digital brain.[226.81] [226.81][S02]And it's not actually that different from how[229.8] [229.8][S02]current large language models or multimodal models[232.74] [232.74][S02]are created today.[233.74] [233.74][S02]There's two basic steps that have[235.47] [235.47][S02]been pretty constant throughout many years[238.58] [238.58][S02]in many of the projects that we've worked on,[241.11] [241.11][S02]which we can call the first one pretraining or imitation[244.7] [244.7][S02]learning.[245.52] [245.52][S02]That is, you start with random weights,[248.07] [248.07][S02]you have an algorithm that will try to imitate lots of data[252.98] [252.98][S02]that humans have created to either play a game[256.25] [256.25][S02]or, in this case, all of the internet, all of the knowledge[258.8] [258.8][S02]available to us.[260.22] [260.22][S02]And in that first stage, you just[263.06] [263.06][S02]adapt the weights to try to imitate that data as well[267.41] [267.41][S02]as possible.[268.23] [268.23][S01] And these weights are, essentially[270.68] [270.68][S01]inside each of the neurons is a series of numbers[275.15] [275.15][S01]that kind of describes how it's connected to everything else?[278.1] [278.1][S02] Yeah, so basically there[280.61] [280.61][S02]are units of computations that are neurons.[283.35] [283.35][S02]And the connections between neurons[286.16] [286.16][S02]are what actually you have as weights.[288.63] [288.63][S02]So you can imagine that there's a neuron,[291.9] [291.9][S02]there's a few neurons connected to it.[294.21] [294.21][S02]And you're basically adding all the activations[297.79] [297.79][S02]from the incoming neurons there multiplied by the weights.[302.06] [302.06][S02]And those weights are the only things that move.[304.81] [304.81][S02]And the inputs excite the neurons.[306.92] [306.92][S02]It's pretty much how a brain works with some freedom[311.14] [311.14][S02]of, yeah, creativity.[312.425] [312.425][S01] OK, if we were to do an analogy,[315.22] [315.22][S01]it's almost like you've got the neurons and you're like,[319.39] [319.39][S01]water is flowing through it and the weights[322.48] [322.48][S01]is like the width of the pipes between the neurons?[324.98] [324.98][S02] Yeah, that's right.[326.397] [326.397][S02]And then you can imagine having millions of neurons and billions[332.8] [332.8][S02]or even trillions of pipes.[334.55] [334.55][S02]And that is what we spend most to compute actually[339.79] [339.79][S02]training these models, especially language models,[342.11] [342.11][S02]is in this pretraining or imitating all the data[344.98] [344.98][S02]that we have available to us.[346.61] [346.61][S01] OK, so you've now got this gigantic network[350.02] [350.02][S01]with loads of pipes going between all the neurons.[352.3] [352.3][S01]And that's your imitation phase done.[355.02] [355.02][S01]Next bit, if you were doing, say, AlphaGo or AlphaZero,[360.88] [360.88][S01]you would then get it to play itself.[363.76] [363.76][S02] Yeah, so this model now[365.64] [365.64][S02]is reasonably good at playing moves that look human-like.[369.4] [369.4][S02]So that means, of course, the sentences[373.5] [373.5][S02]are very plausible sentences in English.[376.27] [376.27][S02]Or if it was playing a game, it would click things reasonably[379.89] [379.89][S02]to move pieces on the board and whatnot.[383.07] [383.07][S02]But what this model hasn't done is[384.99] [384.99][S02]learn that these actions yield reward.[389.62] [389.62][S02]That's the bit of reinforcement learning or post-training,[393.07] [393.07][S02]which is the second phase of training.[395.26] [395.26][S02]So you can write a poem by just, hey,[400.12] [400.12][S02]just how does a poem on the internet look like on average?[403.72] [403.72][S02]But then the question is, well, I want only the good poems.[407.46] [407.46][S02]So how can I further adjust these pipes based on a signal[414.0] [414.0][S02]that now, having written now a whole poem,[417.01] [417.01][S02]would give a score of 0 or 1, let's say.[420.85] [420.85][S02]And if it's a mediocre poem, you get a 0.[424.15] [424.15][S02]If it's a good poem, you get a 1.[425.8] [425.8][S02]Again, for a game analogy, which is[427.83] [427.83][S02]what we use reinforcement learning traditionally, if you[431.22] [431.22][S02]win at the game, you get a 1.[432.73] [432.73][S02]If you lose, you get a 0.[433.96] [433.96][S02]And then you further adjust the weights.[437.17] [437.17][S02]But now, instead of imitating humans,[439.39] [439.39][S02]you're just saying forget, I want[441.57] [441.57][S02]to go beyond what humans could do[443.46] [443.46][S02]and try to really get all my poems to be the perfect poem[447.43] [447.43][S02]or all my chess games to be the perfect game.[449.8] [449.8][S02]And in language models, this second phase,[453.7] [453.7][S02]which is reinforcement learning post-training,[455.85] [455.85][S02]tends to be fairly short-lived because we do not have access[458.76] [458.76][S02]to super clean reward as you've won the game[462.18] [462.18][S02]or you lost the game, when you do[464.31] [464.31][S02]self-play in traditional board games, for example.[467.5] [467.5][S01] So once that's done, that's[470.64] [470.64][S01]all the stuff that goes on behind the scenes.[472.53] [472.53][S01]And then you're like, hold it right there.[475.447] [475.447][S02] Yeah.[476.28] [476.28][S01] Stay exactly where you are, everybody.[478.5] [478.5][S01]We're going to take just basically a snapshot[481.16] [481.16][S01]of this entire network, and that is what you actually[485.54] [485.54][S01]get to access as a user.[488.37] [488.37][S02] Yeah, so now this amazing process finished.[492.4] [492.4][S02]These weights are super precious, right?[494.73] [494.73][S02]So this configuration you found, you've[497.39] [497.39][S02]really spent months to finesse it, to tweak everything.[501.03] [501.03][S02]And now you will never move it anymore.[505.04] [505.04][S02]So training is over.[506.37] [506.37][S02]You're not changing the configuration anymore.[508.92] [508.92][S02]You might want to make it super efficient.[510.9] [510.9][S02]So say you find that, oh, look, this neuron is not that useful.[515.565] [515.565][S02]It's not used for anything.[516.69] [516.69][S02]You remove it, so everything becomes faster and cheaper[520.01] [520.01][S02]to run it at scale.[521.46] [521.46][S02]And then as a user, you just get the same weights.[524.73] [524.73][S02]Everyone gets the same weights we've trained.[526.95] [526.95][S02]That's what we call Gemini 1.5 Flash.[530.12] [530.12][S02]That just means a set of weights that[532.27] [532.27][S02]are frozen, will not change, will not[534.73] [534.73][S02]further train or anything.[536.42] [536.42][S02]So those two steps actually pretty much[538.51] [538.51][S02]are identical from AlphaGo to AlphaStar[542.83] [542.83][S02]to current large language models.[545.208] [545.208][S02]And of course, there's details that matter.[547.0] [547.0][S02]And the field has evolved, certainly,[549.35] [549.35][S02]but the principle is pretty much unchanged, actually.[552.67] [552.67][S01] Because under the hood, as it were,[554.75] [554.75][S01]there are differences between--[556.33] [556.33][S01]I don't know, I'm thinking like DQN here,[558.29] [558.29][S01]which was the Atari example, or the types of algorithms[562.3] [562.3][S01]that are used in AlphaGo or then again, in the large language[565.66] [565.66][S01]models, the architecture is different, right?[567.85] [567.85][S02] Yeah, so there's a few components[569.85] [569.85][S02]that go into then what the digital brain is.[573.14] [573.14][S02]One is the architecture.[574.37] [574.37][S02]So there are these neural networks.[577.6] [577.6][S02]Now we have the transformers, which we certainly didn't[581.2] [581.2][S02]have back in the DQN days.[582.86] [582.86][S02]So there's always some sort of breakthroughs[585.37] [585.37][S02]in architectures that are better at learning from the data.[589.6] [589.6][S02]But then from transformers to today,[592.54] [592.54][S02]it's almost all about little tweaks.[595.55] [595.55][S02]Even if you look at AlphaFold, which[597.22] [597.22][S02]also is fueled by a transformer, what the teams do for years[602.14] [602.14][S02]sometimes is just to find little tweaks on,[604.97] [604.97][S02]hey, let's remove this set of neurons,[607.07] [607.07][S02]let's add another layer, let's make this a bit wider.[610.79] [610.79][S02]So the brain shape changes a little bit,[613.79] [613.79][S02]and that makes it or breaks it sometimes,[615.91] [615.91][S02]in terms of the performance achieved.[618.02] [618.02][S01] So if these are all the things that[619.978] [619.978][S01]have been achieved so far, the goal, as I understand it,[624.56] [624.56][S01]is to create more agentic behavior,[626.8] [626.8][S01]to get these things to make autonomous decisions.[630.83] [630.83][S01]How did these help to achieve that end?[634.053] [634.053][S02] Yeah, so let's zoom[635.47] [635.47][S02]in a little bit on the current trend.[637.79] [637.79][S02]We call it large language models, but they're multimodal.[640.49] [640.49][S02]I think we had an episode earlier covering heavily[643.15] [643.15][S02]the multimodality aspect, how good[644.86] [644.86][S02]it is to be able to add an image,[647.15] [647.15][S02]then ask something, a follow-up question, and so on.[649.74] [649.74][S02]So this score, we will still improve it,[653.12] [653.12][S02]this set of weights that do these amazing inferences[658.67] [658.67][S02]about the input.[659.89] [659.89][S02]What's this image about?[661.08] [661.08][S02]What's the user asking?[662.43] [662.43][S02]Can I write a better poem?[664.02] [664.02][S02]Can I just make it longer or whatever?[665.9] [665.9][S02]Like all these interactions we all get to play with these days.[670.55] [670.55][S02]But this is just a component.[673.25] [673.25][S02]And we can think, hey, this is now our CPU.[677.18] [677.18][S02]And we can add more to it, around it.[680.79] [680.79][S02]What if the model could go off and do research for you,[685.58] [685.58][S02]for example?[686.49] [686.49][S02]One example, we were already thinking[688.61] [688.61][S02]about that back in the day, I could ask a model, a language[691.85] [691.85][S02]model or a visual language model to learn[695.24] [695.24][S02]to play the game of StarCraft.[697.07] [697.07][S02]That's a very different approach to say, create one agent that[700.25] [700.25][S02]does play the game.[701.36] [701.36][S02]In this other example, it could go online, watch videos[706.07] [706.07][S02]about the game.[707.28] [707.28][S02]It could, of course, download the game[709.25] [709.25][S02]to start interacting with it to learn, oh, yeah, I know,[713.06] [713.06][S02]I get it.[714.5] [714.5][S02]Do research online, go to forums, read the forums.[717.92] [717.92][S02]Go play and figure out that it's weak at this thing[721.4] [721.4][S02]and improve and so on.[722.73] [722.73][S02]And after, literally, it could be weeks,[725.27] [725.27][S02]it sends you an email that says, I now know how to play the game.[728.94] [728.94][S02]Let's play.[730.14] [730.14][S02]That's not a reality that's that far away.[732.57] [732.57][S02]But these models all of a sudden actually do something,[737.07] [737.07][S02]take some actions, and learn anything new[739.76] [739.76][S02]that is available to them.[741.18] [741.18][S02]And that's pretty powerful to think about.[744.03] [744.03][S02]It's what pushes the generality the most.[746.55] [746.55][S02]And that's what makes the AGI, as many people call it,[751.76] [751.76][S02]feel closer.[752.748] [752.748][S01] So if I understand it correctly then,[754.79] [754.79][S01]it's almost like the stuff that we[756.44] [756.44][S01]have at the moment, the large language models,[757.98] [757.98][S01]the multimodal models, whatever you want to call them,[760.23] [760.23][S01]that's the central core.[762.18] [762.18][S01]But the next step is that you build stuff[765.13] [765.13][S01]on top of that central core, that it can go off[767.92] [767.92][S01]and take off the stabilizers and go off and do its own thing.[771.972] [771.972][S02] Yeah, exactly.[773.18] [773.18][S02]If it has access to all the knowledge[775.75] [775.75][S02]and it can use its time to do some proper research,[780.71] [780.71][S02]write hypotheses, write some code[782.32] [782.32][S02]and so on, and take its time to really answer very, very, very[785.92] [785.92][S02]complex questions, then, yeah, the possibilities[789.89] [789.89][S02]now have broadened quite drastically.[792.32] [792.32][S02]Although of course, we're not going[794.11] [794.11][S02]to need that for everything.[795.53] [795.53][S02]I mean, if we ask a question like, hey, I like rice.[800.24] [800.24][S02]What should I prepare tonight?[801.53] [801.53][S02]Probably no need to do a very deep dive into thinking or just[806.08] [806.08][S02]going off for three weeks.[807.74] [807.74][S02]Then you'll probably not be very happy about the waiting time,[810.595] [810.595][S02]right?[811.6] [811.6][S02]But I think to push the frontier,[815.27] [815.27][S02]you're giving a digital body to the computer so it can not[819.04] [819.04][S02]only just think and give an instruction or a word output,[823.52] [823.52][S02]but it can also go off and do things online[826.98] [826.98][S02]or on documents that you might upload or whatever.[830.5] [830.5][S02]And ask very, very complex questions and personalized[832.89] [832.89][S02]to you, et cetera, et cetera.[834.13] [834.13][S01] I like that idea, this central core and then[836.07] [836.07][S01]you're giving it a digital body.[837.4] [837.4][S01]You've got the electric brain, and now you're[839.275] [839.275][S01]giving it a digital body.[840.43] [840.43][S02] Exactly.[841.05] [841.05][S01] OK, so in terms of the electric brain[843.092] [843.092][S01]and in terms of this core, this processor,[844.858] [844.858][S01]let me just ask you a bit about that.[846.4] [846.4][S01]I guess we should be thinking about Gemini here, right?[849.013] [849.013][S02] Mm-hmm.[849.93] [849.93][S01] Which is essentially what we're talking about,[851.32] [851.32][S01]the multimodal model that you guys have.[853.78] [853.78][S01]I know that one of the big ideas for large models[856.95] [856.95][S01]was just to scale it up, to get them bigger and bigger[859.26] [859.26][S01]and bigger and bigger and bigger.[860.74] [860.74][S01]Do you think that the results that we've seen from scaling[864.33] [864.33][S01]have plateaued by now?[866.26] [866.26][S02] Yeah, it's a very important question.[868.88] [868.88][S02]We have studied how as you make the models larger,[872.83] [872.83][S02]that is how many neurons literally these models have,[877.33] [877.33][S02]how do they become better at certain tasks[881.16] [881.16][S02]that we have clear metrics from the whole machine[883.68] [883.68][S02]learning community?[884.53] [884.53][S02]For example, one that is very simple to understand[887.52] [887.52][S02]is machine translation.[888.9] [888.9][S02]So how good the models are at translating[892.17] [892.17][S02]between two languages as you scale,[894.64] [894.64][S02]as you go from millions to billions[897.03] [897.03][S02]to potentially trillions of neurons,[900.03] [900.03][S02]you can see the performance keep improving.[902.64] [902.64][S02]Now, even when you do those studies,[905.83] [905.83][S02]one trick is that it looks linear,[909.43] [909.43][S02]but you have to plot logarithmic axis.[913.26] [913.26][S02]What that means in lay terms is that, let's say[917.58] [917.58][S02]from the last three years, we had some improvement,[920.83] [920.83][S02]you shouldn't expect the same improvement[922.92] [922.92][S02]in the next three years.[924.15] [924.15][S02]It's actually exponentially hard to get there.[926.74] [926.74][S02]So that means the compute investment, which[930.0] [930.0][S02]of course, also advances at the super linear rate,[932.74] [932.74][S02]but perhaps not as good as these sort of trends suggest,[937.485] [937.485][S02]you would just see some diminishing returns.[940.05] [940.05][S02]Because simply scaling the x-axis,[943.08] [943.08][S02]the number of parameters, you need to go 10x to see the same[947.03] [947.03][S02]improvement.[948.03] [948.03][S02]And that just creates some pressure to,[951.71] [951.71][S02]hey, maybe we can't scale as much[954.78] [954.78][S02]and we need to think about other ways[956.39] [956.39][S02]to scale to make the models better.[957.965] [957.965][S01] The example I give to my students[959.84] [959.84][S01]is like, if you've got a room that's really messy,[962.55] [962.55][S01]the first 10 minutes that you spend tidying,[964.74] [964.74][S01]it's going to make a massive difference.[966.03] [966.03][S01]You pick up all the dirty plates,[967.05] [967.05][S01]put away all the dirty washing, fine.[968.91] [968.91][S01]But once you're like 7 hours in, that extra 10 minutes,[972.69] [972.69][S01]it's not going to make any difference at all.[974.698] [974.698][S01]And that's essentially where we are, right?[976.49] [976.49][S02] Yeah, that's exactly a very good analogy.[978.9] [978.9][S02]And in fact, that analogy can even[980.63] [980.63][S02]apply to then the performance of the models.[982.92] [982.92][S02]Even if you have extremely good performance,[985.47] [985.47][S02]if you want these models to be 100% factual,[988.962] [988.962][S02]they will never make something up--[990.42] [990.42][S02]we know that if you probe them, you can make them say some[993.17] [993.17][S02]things that are not real--[995.27] [995.27][S02]even that last mile also is super hard,[998.43] [998.43][S02]which creates some interesting challenges to deploy them[1001.45] [1001.45][S02]at scale.[1002.21] [1002.21][S01] So OK, I hear what you're[1003.16] [1003.16][S01]saying about how there's diminishing returns[1004.993] [1004.993][S01]in all of this.[1005.62] [1005.62][S01]But in terms of how you make these things better,[1008.9] [1008.9][S01]how you make these models better,[1010.37] [1010.37][S01]is it just data, computational power, and size?[1014.42] [1014.42][S01]Are those the only things, the levers that you have to pull?[1017.14] [1017.14][S02] Yeah, so certainly,[1019.54] [1019.54][S02]if you froze the architecture, let's say[1022.93] [1022.93][S02]for the next year, no innovation,[1025.19] [1025.19][S02]we just scale because there's better hardware coming out--[1027.607] [1027.607][S01] Just make it bigger, yeah.[1029.19] [1029.19][S02] --make it bigger, that certainly[1031.329] [1031.329][S02]would have a trend that would look OK.[1033.74] [1033.74][S02]But what's happened, and certainly in Gemini,[1036.88] [1036.88][S02]we have other innovations, other tricks, techniques,[1042.91] [1042.91][S02]details from how to order the data that you present[1048.099] [1048.099][S02]the model with, to the details of the architecture,[1051.62] [1051.62][S02]to how to run the training process, how long to run it for.[1056.14] [1056.14][S02]What kind of data do we actually present the model?[1059.34] [1059.34][S02]How do we filter?[1060.6] [1060.6][S02]Do we present more data that's high quality, less data that's[1063.29] [1063.29][S02]low quality?[1063.96] [1063.96][S02]All sorts of different what we call hyperparameters[1068.25] [1068.25][S02]and, of course other algorithmic advances,[1070.71] [1070.71][S02]we also investigate fairly carefully.[1073.55] [1073.55][S02]Because the process of training a model is expensive.[1076.65] [1076.65][S02]So we need to be extremely careful with piling up[1080.15] [1080.15][S02]innovation so that eventually when we are ready,[1083.22] [1083.22][S02]we have enough innovation.[1084.63] [1084.63][S02]And also probably we'll have a better scale[1087.59] [1087.59][S02]to run for the next iteration of models.[1090.12] [1090.12][S02]We run it, and then we get algorithmic, not[1092.87] [1092.87][S02]only breakthroughs through data and compute.[1095.56] [1095.56][S01] I guess the other thing about this scaling stuff[1098.06] [1098.06][S01]is that there's no limit, really, to the number of nodes[1101.63] [1101.63][S01]that you can put in.[1102.84] [1102.84][S01]Maybe there's no limit, in theory,[1105.308] [1105.308][S01]to the computational power that you put in.[1107.1] [1107.1][S01]But there is a limit to the data that you can put in.[1109.08] [1109.08][S01]There's a limit to the number of human words that are out there.[1111.75] [1111.75][S02] Good point.[1112.833] [1112.833][S02]So I think there is a limit on the nodes.[1115.91] [1115.91][S02]Because how you scale these models is,[1118.07] [1118.07][S02]well, they don't fit on one single chip, hardware chip.[1120.59] [1120.59][S02]So now you have a mesh of chips that are communicating.[1123.8] [1123.8][S02]There's certain limits like speed of light,[1125.775] [1125.775][S02]et cetera, et cetera.[1126.65] [1126.65][S01] [LAUGHS][1127.405] [1127.405][S02] So there starts to be[1128.905] [1128.905][S02]a time where the efficiency of training such a big model[1131.8] [1131.8][S02]is also just not worth it, even from a utilization[1135.01] [1135.01][S02]of the hardware at your disposal, but very good point.[1138.585] [1138.585][S02]The other bit that is critical on this pretraining, imitate[1142.57] [1142.57][S02]all the data, is that we do not have what[1145.06] [1145.06][S02]we call infinite data regime.[1147.67] [1147.67][S02]There's finite data.[1149.48] [1149.48][S02]And so as soon as the models need to--[1152.86] [1152.86][S02]you can think, well, let's train on all the data.[1155.183] [1155.183][S02]If you want to train--[1156.1] [1156.1][S01] Everything humans have ever read--[1157.21] [1157.21][S02] Everything.[1157.7] [1157.7][S01] --or written.[1158.14] [1158.14][S02] All of the internet.[1159.598] [1159.598][S02]So we're just starting to think, OK, we're running out of data.[1163.16] [1163.16][S02]There are techniques like synthetic data.[1165.2] [1165.2][S02]Can we write or rewrite existing data in many different ways?[1169.45] [1169.45][S02]I mean, languages would be obvious ways to think, hey,[1171.7] [1171.7][S02]you could write the internet.[1173.84] [1173.84][S02]It's mostly in English, I mean, 60%.[1175.75] [1175.75][S02]I don't know what's the exact percentage.[1177.81] [1177.81][S02]But there are ways to rewrite the same knowledge[1180.8] [1180.8][S02]in different ways.[1181.98] [1181.98][S02]We're exploring those.[1183.42] [1183.42][S02]That's kind of a research area that many people[1185.6] [1185.6][S02]are starting to invest.[1187.02] [1187.02][S02]Because if you run out of data, the scaling laws[1190.85] [1190.85][S02]punish you even more.[1192.5] [1192.5][S01] So for example, then, you[1194.81] [1194.81][S01]could get Gemini to write its own version of the internet,[1199.05] [1199.05][S01]and then use that to train a new version of Gemini?[1201.428] [1201.428][S02] Yes.[1202.22] [1202.22][S01] Is there a danger, though,[1204.09] [1204.09][S01]that if you start feeding in the output of the same model,[1208.89] [1208.89][S01]that you can end up creating these little, well, unhelpful[1212.45] [1212.45][S01]feedback loops?[1213.87] [1213.87][S02] They certainly can[1215.45] [1215.45][S02]do some interesting experiments to test ideas[1218.9] [1218.9][S02]like this one you just mentioned.[1220.74] [1220.74][S02]And indeed, that is, on the surface, not a good idea.[1225.74] [1225.74][S02]The model suffers if you just ask it to recreate[1229.55] [1229.55][S02]all of the internet.[1230.66] [1230.66][S02]And indeed, a priori from information[1233.82] [1233.82][S02]an content point of view, look, this data set[1236.62] [1236.62][S02]has the information that it has, how could you[1239.56] [1239.56][S02]create new information?[1241.05] [1241.05][S02]I don't know.[1241.93] [1241.93][S02]These ideas might help a little bit[1244.735] [1244.735][S02]because there's machine learning deficiencies[1246.61] [1246.61][S02]that we're not at that fundamental ability[1250.06] [1250.06][S02]to extract all the information truly from the internet.[1252.823] [1252.823][S02]We have good algorithms, but they're not perfect.[1254.865] [1254.865][S01] Well, yeah.[1255.67] [1255.67][S02] So we'll see, yeah.[1256.55] [1256.55][S01] I guess I just want to think about that a little bit[1257.89] [1257.89][S01]more because it's a really interesting idea.[1259.18] [1259.18][S01]Because of course, naively, if you did it without thinking,[1261.8] [1261.8][S01]then it's like, the new version would have the biases in it.[1267.06] [1267.06][S01]And then the new version on top of that would be more biased.[1270.005] [1270.005][S01]And you'd end up sort of spiraling away[1271.63] [1271.63][S01]from the original human one.[1273.44] [1273.44][S01]But then what you're saying is as[1274.99] [1274.99][S01]though in the original human internet[1277.57] [1277.57][S01]is sort of embedded these conceptual connections.[1281.42] [1281.42][S01]And if you can extract those--[1282.92] [1282.92][S01]I'm thinking almost like E equals mc squared.[1285.13] [1285.13][S01]If you can find the E equals mc squared for human concepts[1290.29] [1290.29][S01]and then just generate new data using that alone,[1294.45] [1294.45][S01]then that seems more realistic.[1295.95] [1295.95][S02] Yeah, exactly.[1297.158] [1297.158][S02]And I think that's where you start hitting--[1299.57] [1299.57][S02]I mean, are these language models[1301.76] [1301.76][S02]just repeating what's online and not[1304.28] [1304.28][S02]being able to create anything new?[1306.06] [1306.06][S02]Or are they learning a world model[1308.45] [1308.45][S02]truly that you can then from the principles it extracts, possibly[1313.55] [1313.55][S02]generalize beyond what the data has?[1315.69] [1315.69][S02]And under the more optimistic version,[1318.0] [1318.0][S02]which I tend to believe more, we can push the limits of data[1322.82] [1322.82][S02]little bit more than the current limits that we have.[1325.98] [1325.98][S02]That being said, there are some data sources[1329.51] [1329.51][S02]that we haven't quite seen a breakthrough, like video data.[1334.35] [1334.35][S02]There's a lot of it.[1335.43] [1335.43][S02]And we haven't quite seen a moment[1338.12] [1338.12][S02]of take all the video data where you probably[1341.54] [1341.54][S02]can derive a lot of knowledge, a lot of laws of physics,[1344.91] [1344.91][S02]a lot of how the world works, even if there are[1347.12] [1347.12][S02]no words associated with the videos necessarily,[1350.28] [1350.28][S02]and extract that knowledge.[1351.96] [1351.96][S02]Even that I don't think we tapped into that source.[1354.58] [1354.58][S01] And it doesn't work that way?[1356.44] [1356.44][S02] I mean--[1356.86] [1356.86][S01] Or you don't know?[1358.11] [1358.11][S02] Yeah, it feels like it should,[1360.59] [1360.59][S02]even how we learn.[1363.558] [1363.558][S02]There's some language learning in the early days,[1365.6] [1365.6][S02]but we learn by also observing three dimensions and so[1368.98] [1368.98][S02]on and so forth.[1369.65] [1369.65][S02]So there probably is more knowledge[1373.6] [1373.6][S02]that we haven't extracted.[1375.43] [1375.43][S02]What obviously we've gotten pretty well at,[1378.47] [1378.47][S02]and you can see by testing the models,[1380.12] [1380.12][S02]is connecting the concepts present in the video.[1384.08] [1384.08][S02]And then you can do amazing things like, hey,[1386.03] [1386.03][S02]take this full hour video and just extract me[1388.9] [1388.9][S02]three interesting moments.[1390.53] [1390.53][S02]But the model itself, has it actually used that--[1393.995] [1393.995][S01] Oh, yeah.[1394.87] [1394.87][S02] --information directly?[1397.16] [1397.16][S02]Probably not.[1397.76] [1397.76][S01] Oh, I like this so much.[1399.26] [1399.26][S01]We were talking to Jeff about with multimodal models,[1402.14] [1402.14][S01]if you get these models to just watch[1404.8] [1404.8][S01]all of the videos that have ever been created,[1407.03] [1407.03][S01]can it quite literally extract what gravity means as a concept?[1411.513] [1411.513][S01]But what you're describing here, if I understand it,[1413.68] [1413.68][S01]is that at the moment, it can tell you what's in the video[1416.79] [1416.79][S01]that it's seen, but it can't then say,[1419.16] [1419.16][S01]and E equals mc squared.[1421.81] [1421.81][S01]Or if you showed it pictures of the night sky,[1424.33] [1424.33][S01]it wouldn't suddenly be able to predict the planetary motion[1427.53] [1427.53][S01]in the same way that human astronomers did.[1429.692] [1429.692][S02] Yeah, exactly.[1430.9] [1430.9][S02]The shortcut we're taking here is[1432.275] [1432.275][S02]that the data we train when we train on images and/or videos,[1437.23] [1437.23][S02]we almost always have a text representation[1442.26] [1442.26][S02]associated with that modality.[1443.77] [1443.77][S02]So it could be a caption explaining[1446.04] [1446.04][S02]what this image has or this video has[1447.96] [1447.96][S02]and so on and so forth.[1449.35] [1449.35][S02]And, of course, then it's incredible, right?[1452.86] [1452.86][S02]You can put a picture of homework[1454.71] [1454.71][S02]and with a little drawing conceptual[1456.6] [1456.6][S02]and it will connect and do quite a lot of good logic just based[1460.56] [1460.56][S02]on that.[1461.32] [1461.32][S02]But what I'm saying here is, could I just[1464.4] [1464.4][S02]take videos, no language, and then train[1467.34] [1467.34][S02]a model to then understand what's happening?[1470.98] [1470.98][S02]Maybe even, in a way, derive a language--[1473.91] [1473.91][S02]it's obviously not going to be our language--[1476.88] [1476.88][S02]and extract those concepts.[1478.99] [1478.99][S02]And that has not happened.[1481.63] [1481.63][S02]And it probably will.[1486.18] [1486.18][S01] Just going back to what[1488.07] [1488.07][S01]you said at the beginning about those two phases[1490.29] [1490.29][S01]to basically all the models that the DeepMind have built.[1493.507] [1493.507][S02] Yeah.[1494.34] [1494.34][S01] The imitation phase, which[1496.05] [1496.05][S01]is what we've been talking about right[1497.633] [1497.633][S01]here, but then the reinforcement learning phase on top.[1501.1] [1501.1][S01]And I know that AlphaGo and AlphaZero and many more[1505.29] [1505.29][S01]got better by playing themselves.[1507.73] [1507.73][S01]Does that apply here as well?[1510.16] [1510.16][S02] Yeah, that's one of the main open challenges--[1514.11] [1514.11][S02]scaling, not only pretraining but postraining or reinforcement[1518.04] [1518.04][S02]learning.[1518.926] [1518.926][S02]So the beauty about reinforcement learning in games[1522.9] [1522.9][S02]is that there is a set of rules that are coded.[1526.96] [1526.96][S02]And if you've won, you know you've won.[1529.34] [1529.34][S02]There is kind of a program that if you play chess[1532.22] [1532.22][S02]and you've won, it will check everything.[1534.66] [1534.66][S02]OK, that's a check mate.[1535.998] [1535.998][S02]Congratulations, you've won the game.[1537.54] [1537.54][S01] A clear metric of success.[1538.68] [1538.68][S02] A clear metric.[1540.11] [1540.11][S02]Now, in language, much trickier.[1544.232] [1544.232][S02]Is this a better poem than this?[1546.21] [1546.21][S02]Good luck discussing this, even amongst us.[1548.28] [1548.28][S02]So the generality again makes computing exactness very hard.[1556.44] [1556.44][S02]Is this a better summary of the movie?[1558.39] [1558.39][S02]Is this the most interesting bit of this video?[1561.21] [1561.21][S02]It's very hard to quantify.[1562.92] [1562.92][S02]But we can try, and we do try.[1565.17] [1565.17][S02]You train a model, and based on some human preferences,[1569.27] [1569.27][S02]roughly you just say, OK, try to now generalize.[1573.06] [1573.06][S02]So if I ask a model to criticize its own output,[1575.91] [1575.91][S02]it's not going to do that bad.[1577.62] [1577.62][S02]It's going to be good maybe 80% of the time, which[1581.33] [1581.33][S02]is not terrible.[1582.77] [1582.77][S02]But it can give you some signal.[1584.64] [1584.64][S02]But at that point, you start saying, well,[1586.68] [1586.68][S02]now you climb this metric.[1588.62] [1588.62][S02]You have this imperfect way to assess performance.[1591.98] [1591.98][S02]But now we're going to start training[1595.63] [1595.63][S02]against this reward that is not perfect.[1599.11] [1599.11][S02]What the model is going to do is exploit[1601.99] [1601.99][S02]the weaknesses of the reward.[1605.2] [1605.2][S02]Maybe using the chess example, imagine that I had a bug,[1609.14] [1609.14][S02]and if a pawn is in a certain position, you always win.[1612.7] [1612.7][S02]And it's a position that no one would play ever.[1616.43] [1616.43][S02]So maybe no one knows this exists.[1618.26] [1618.26][S02]But now you ask an algorithm to please[1621.49] [1621.49][S02]explore everything and try to discover[1623.44] [1623.44][S02]how to win at this game.[1625.04] [1625.04][S02]All of a sudden, you're going to find, hey, oh,[1627.53] [1627.53][S02]if I move the first pawn to this position,[1629.9] [1629.9][S02]no one plays this opening, You've.[1631.78] [1631.78][S02]Won the game.[1632.69] [1632.69][S02]Certainly, you the algorithm has nailed the game.[1636.74] [1636.74][S02]And then a researcher goes and sees how you play chess,[1639.49] [1639.49][S02]and it's just terrible.[1640.76] [1640.76][S01] Naughty AI, basically[1642.19] [1642.19][S01]that's what we're talking about.[1642.98] [1642.98][S02] So that's the challenge.[1644.87] [1644.87][S02]Basically you're finding exploits[1646.44] [1646.44][S02]rather than really learning what a good poem means truthfully.[1651.465] [1651.465][S01] Can you not just add in another player?[1653.59] [1653.59][S01]So add in another model, which is the ultimate arbiter?[1660.36] [1660.36][S02] Good suggestion, but then the problem[1662.79] [1662.79][S02]is, how do you train that model?[1665.782] [1665.782][S02]We have only a finite notion of what's[1669.3] [1669.3][S02]a good poem by some experts that we might ask, hey, compare[1672.51] [1672.51][S02]these two poems and so on.[1673.78] [1673.78][S02]So there's just a limited amount of data we[1677.19] [1677.19][S02]have to train these arbiters.[1679.57] [1679.57][S02]So the ground truth might be to ask someone[1683.01] [1683.01][S02]that is the expert, of course.[1685.327] [1685.327][S02]And if we could, we would.[1686.41] [1686.41][S02]But that's not scalable.[1688.12] [1688.12][S02]Then imagine how slow it would be to say,[1691.06] [1691.06][S02]OK, I have a parameter update that in three seconds, I found.[1696.88] [1696.88][S02]Now please review these 10,000 things by an expert.[1700.08] [1700.08][S02]Because that's the source of truth.[1702.98] [1702.98][S02]And we don't have enough data to train a good enough reward[1705.44] [1705.44][S02]model.[1705.94] [1705.94][S02]So again, there are some ideas.[1708.03] [1708.03][S02]But intuitively, you clearly understand,[1711.82] [1711.82][S02]but the problem is we don't have access to the ground truth.[1715.103] [1715.103][S01] That's it.[1716.02] [1716.02][S01]It's like you're feeling around in the dark with oven gloves on.[1718.83] [1718.83][S02] Yeah.[1719.25] [1719.25][S01] [LAUGHS][1719.83] [1719.83][S02] Yeah.[1720.085] [1720.085][S01] [LAUGHS] You're not even completely sure that there[1722.85] [1722.85][S01]are solid objects--[1723.9] [1723.9][S02] Yeah.[1723.99] [1723.99][S01] --to grab onto.[1725.115] [1725.115][S01]OK, so if that's the core then, that's the electronic brain,[1729.96] [1729.96][S01]and now we're building the digital body, what kind[1734.34] [1734.34][S01]of capabilities do you want that digital body to have,[1736.8] [1736.8][S01]like reasoning, for example?[1738.193] [1738.193][S01]Because there's been quite a lot of work[1739.86] [1739.86][S01]on that too, hasn't there?[1741.22] [1741.22][S02] Yeah, so when you start thinking, well,[1743.76] [1743.76][S02]what are the main surfaces that we would[1748.71] [1748.71][S02]be able to give these models limited access to so they can[1752.88] [1752.88][S02]see beyond what's in their weights, which are frozen,[1756.42] [1756.42][S02]to be able to gather knowledge or do maybe something a bit more[1760.5] [1760.5][S02]complex than just predicting the next word from just what they[1764.57] [1764.57][S02]have in context, plus what they have in their weights?[1768.12] [1768.12][S02]And so obvious things that come to mind[1771.53] [1771.53][S02]is giving them access to a search engine.[1774.09] [1774.09][S02]That's what we do very well at Google.[1777.38] [1777.38][S02]Another one is to give them the ability[1780.02] [1780.02][S02]to run the code they write.[1781.885] [1781.885][S02]And then, of course, maybe even more broadly, that[1785.33] [1785.33][S02]could be more general is giving them the ability[1788.57] [1788.57][S02]to maybe interact with a browser that has access to the internet.[1792.77] [1792.77][S02]With all of these, you always have[1794.66] [1794.66][S02]to be careful to sand box, that just means protect[1798.26] [1798.26][S02]these environments so that the models, even if they're not[1800.78] [1800.78][S02]that advanced, wouldn't do something that is unintended.[1804.32] [1804.32][S02]So there's the whole safety aspect of this[1807.06] [1807.06][S02]that as you move beyond the model,[1809.22] [1809.22][S02]it starts to be quite interesting.[1811.17] [1811.17][S02]But if we're just dreaming what would be possible,[1815.1] [1815.1][S02]by having these tools available to the models, all of a sudden[1818.99] [1818.99][S02]they can start doing much more advanced things[1822.6] [1822.6][S02]beyond what was the training corpus that we used at the time.[1828.7] [1828.7][S02]They can rely on the latest news to explain us[1832.9] [1832.9][S02]or to summarize what was the main thing[1835.36] [1835.36][S02]yesterday that happened.[1836.9] [1836.9][S02]All these kind of things, you need to give them these tools.[1840.5] [1840.5][S01] OK, so how does reasoning fit into all of this[1843.01] [1843.01][S01]then?[1843.65] [1843.65][S02] Yeah, reasoning is interesting, right?[1845.858] [1845.858][S02]So what I described could be summarized as,[1848.86] [1848.86][S02]hey, I want to know what happened yesterday,[1852.46] [1852.46][S02]then I could just say, look, maybe personalize it[1856.158] [1856.158][S02]a little bit.[1856.7] [1856.7][S02]So I could describe in words, I could say, hey hey, model,[1860.32] [1860.32][S02]I'm Oriol, I'm interested in this and that.[1862.85] [1862.85][S02]My political views are this or that.[1864.82] [1864.82][S02]Give me a positive take on the news yesterday.[1867.53] [1867.53][S02]And the model would probably search, retrieve all the news,[1871.37] [1871.37][S02]and then, given what I asked it to do,[1873.5] [1873.5][S02]just do it in a way that I like and I find it enjoyable.[1876.8] [1876.8][S02]Maybe if I don't like it, I can even go[1878.47] [1878.47][S02]and then say, I didn't like this or this is not a good joke.[1881.62] [1881.62][S02]And then we could iterate a little bit in a conversation.[1884.62] [1884.62][S02]Now, reasoning is a bit of a different axis of scaling.[1890.95] [1890.95][S02]And so you could imagine the model[1894.99] [1894.99][S02]deciding what kind of intermediate steps[1897.51] [1897.51][S02]to do to give me a better answer.[1899.8] [1899.8][S02]So imagine there's 100 news outlets[1903.24] [1903.24][S02]that Google search retrieves.[1905.4] [1905.4][S02]Maybe the model decides, hey, I'm[1907.32] [1907.32][S02]just not going to read this and just[1909.18] [1909.18][S02]try to summarize it all at once, I'm[1911.22] [1911.22][S02]going to summarize each of the 100 articles first.[1914.71] [1914.71][S02]So that means the model decided I'm[1917.25] [1917.25][S02]going to write a summary for each of the 100 pages.[1920.86] [1920.86][S02]I'm going to write it not to the user, to myself.[1924.16] [1924.16][S02]And now it has hundreds summaries.[1926.773] [1926.773][S02]And maybe the next step it decides to do[1928.44] [1928.44][S02]is, I'm going to group these by topics.[1932.19] [1932.19][S02]Then it decides one of the articles looks suspicious,[1935.74] [1935.74][S02]so maybe it goes online and checks if in any forums[1939.06] [1939.06][S02]someone discusses like, oh, this might[1940.91] [1940.91][S02]be not truthful because of the author and so on and so forth.[1945.03] [1945.03][S02]So it can do a lot of steps to do research.[1948.48] [1948.48][S02]And it could do this for quite a while.[1951.06] [1951.06][S02]And only when the model says, well, I[1953.99] [1953.99][S02]think now I have a much better quality answer,[1956.87] [1956.87][S02]then it will give you the few-word summary.[1960.48] [1960.48][S02]But now it had all this time to do much more processing[1963.62] [1963.62][S02]on the information that was available to it.[1966.65] [1966.65][S02]And that inference time compute, we[1970.88] [1970.88][S02]hope that the more time we give to the model,[1973.98] [1973.98][S02]the better it's going to summarize[1975.43] [1975.43][S02]the news, the better it's going to write a poem,[1977.43] [1977.43][S02]the better it's going to, of course, do math.[1979.305] [1979.305][S02]But that's certainly another axis of scaling[1981.41] [1981.41][S02]which we're starting to quite unlock, we hope to unlock.[1985.23] [1985.23][S02]And again, we'll break a bit of the scaling laws and the limits[1988.85] [1988.85][S02]that we see in pure pretraining.[1991.44] [1991.44][S01] Does this also include planning?[1994.656] [1994.656][S01]Could it look at your calendar, work out when your payday was,[1999.81] [1999.81][S01]maybe know that the January sales are coming up soon,[2002.74] [2002.74][S01]and tell you to postpone booking your holiday for a few days?[2008.39] [2008.39][S02] That can get very complex,[2010.19] [2010.19][S02]but of course, when you factor in things like personalization[2014.23] [2014.23][S02]and when to do things because of all the other things[2017.83] [2017.83][S02]that are ongoing, you have more sources of information.[2021.89] [2021.89][S02]You need to collect them and then give the best answer.[2025.73] [2025.73][S02]And it stops being what color is the sky, which[2029.02] [2029.02][S02]is not that simple to answer.[2030.89] [2030.89][S02]I was thinking about that example.[2032.42] [2032.42][S02]We had a very early paper where we have that example,[2035.363] [2035.363][S02]as, oh, something that language models can do.[2037.28] [2037.28][S02]Amazing, right?[2037.94] [2037.94][S02]You don't program them to answer, but they answer.[2040.28] [2040.28][S02]But then, actually even the answer[2042.61] [2042.61][S02]is quite nuanced if you start thinking, oh, yeah,[2045.83] [2045.83][S02]planets, and what time of the day?[2048.38] [2048.38][S02]Is it cloudy or not?[2049.4] [2049.4][S02]So the thinking and the planning,[2052.01] [2052.01][S02]yeah, that's something that these models can do.[2059.472] [2059.472][S01] I'm reminded, I had a conversation with Dennis[2061.889] [2061.889][S01]probably back in 2019.[2063.489] [2063.489][S01]And he was talking about the Kahneman and Tversky[2067.32] [2067.32][S01]ideas of how the human brain has almost two systems of thinking,[2071.409] [2071.409][S01]the sort of quick, instinctive, intuition based,[2074.02] [2074.02][S01]and then the much slower, calculated, the way that you do[2077.88] [2077.88][S01]maths and chess.[2079.0] [2079.0][S01]And Demis was saying that that second one[2082.679] [2082.679][S01]has been traditionally easier for us to do with computers.[2087.06] [2087.06][S01]But now we're seeing the much quicker instinctive stuff.[2091.12] [2091.12][S01]But you're sort of talking about putting the two together, right?[2094.65] [2094.65][S02] Yeah, right.[2096.382] [2096.382][S02]Probably what Demis was talking about[2097.95] [2097.95][S02]is systems too, which is indeed one that you reflect a bit more.[2102.36] [2102.36][S02]And in games it's very clear.[2103.75] [2103.75][S02]You just could say, oh, this move feels right, you just move.[2106.99] [2106.99][S02]But if you think and ponder, you might get to a better move.[2111.96] [2111.96][S02]The challenge is that, now because we[2115.38] [2115.38][S02]are in such general direction-- these models can do[2118.73] [2118.73][S02]anything, anything, literally.[2120.12] [2120.12][S02]You just do whatever you want, upload[2122.15] [2122.15][S02]an image, talk about the news.[2124.14] [2124.14][S02]So what it means to have this deeper[2129.05] [2129.05][S02]thinking is so domain specific that,[2131.81] [2131.81][S02]how are you going to do that?[2133.11] [2133.11][S02]And there's a few answers.[2135.03] [2135.03][S02]But the one I like is like, well, these models[2137.81] [2137.81][S02]are very general.[2138.98] [2138.98][S02]To add the ability to think on top[2141.53] [2141.53][S02]of a very general set of capabilities,[2143.97] [2143.97][S02]you probably need a general way to think.[2146.13] [2146.13][S02]And so you use the model itself to generate how[2151.19] [2151.19][S02]it should think about anything.[2152.85] [2152.85][S02]And the model will come up with, oh, I'm[2155.42] [2155.42][S02]going to summarize each article.[2156.81] [2156.81][S02]I'm going to do this and that and that.[2158.435] [2158.435][S02]And it is not us programming it.[2160.08] [2160.08][S02]That's a very deep insight.[2161.9] [2161.9][S02]Now is it the only way to do it and is it the best way to do it?[2165.96] [2165.96][S02]Early days.[2166.592] [2166.592][S01] Yeah.[2167.3] [2167.3][S02] Five years.[2167.91] [2167.91][S02]We'll see.[2168.02] [2168.02][S01] [LAUGHS] Exactly, I'll talk to you in 2029.[2170.065] [2170.065][S02] Yeah.[2170.55] [2170.55][S01] OK, I'm thinking now though[2172.175] [2172.175][S01]also about lots of the things that felt very important back,[2176.98] [2176.98][S01]five years ago.[2178.45] [2178.45][S01]And a lot of it was about inspiration from neuroscience.[2181.73] [2181.73][S01]So I suppose in a way here, you're[2183.22] [2183.22][S01]talking about planning and reasoning.[2184.85] [2184.85][S01]But memory was the other really big one.[2187.34] [2187.34][S01]And has that come through?[2189.47] [2189.47][S01]People talk about long context and short context a lot.[2192.53] [2192.53][S01]I suppose that is working memory in a way, isn't it?[2195.53] [2195.53][S02] Yeah, there's techniques[2199.03] [2199.03][S02]that are out there that you can apply to a language model.[2202.67] [2202.67][S02]There are at the very least three.[2206.02] [2206.02][S02]And they're reasonably simple to explain.[2209.512] [2209.512][S02]The first way in which we have a system that[2212.71] [2212.71][S02]memorizes all of the internet is by literally doing[2216.19] [2216.19][S02]the pretraining step.[2217.51] [2217.51][S02]That's literally a memorization step[2220.3] [2220.3][S02]in a particular format, which is we have these weights,[2223.55] [2223.55][S02]they're random, and then we assemble them[2226.12] [2226.12][S02]in these amazing architectures.[2228.13] [2228.13][S02]Now, the second level is maybe I explained a little bit[2233.17] [2233.17][S02]how you would give the tool of a search engine such as Google[2236.56] [2236.56][S02]to the model.[2237.61] [2237.61][S02]That, you could claim, is what neuroscientists[2242.86] [2242.86][S02]would call episodic memory, which, as a human,[2246.16] [2246.16][S02]maybe it's like we have these memories from a long time ago.[2250.61] [2250.61][S02]They're not very precise.[2251.75] [2251.75][S02]So they tend to be a bit more fuzzy.[2253.34] [2253.34][S02]If I have to think, oh, what was my first day at Google?[2257.12] [2257.12][S02]I remember bits and pieces, being in a room or someone[2259.99] [2259.99][S02]I met or whatnot.[2260.955] [2260.955][S01] The gist.[2261.83] [2261.83][S02] Yeah, the gist, right?[2263.372] [2263.372][S02]Now, interestingly, these models maybe[2265.18] [2265.18][S02]don't have that limitation.[2266.53] [2266.53][S02]You can literally get an article written many years ago online,[2270.32] [2270.32][S02]and it's going to have all the images, everything[2272.53] [2272.53][S02]will be perfect, reconstructed perfectly.[2275.06] [2275.06][S02]So that second mode called episodic memory, clearly[2278.89] [2278.89][S02]we're seeing that when you integrate especially powerful[2281.56] [2281.56][S02]search engines into our models.[2284.12] [2284.12][S02]And then the third one is what you[2287.11] [2287.11][S02]could call working memory, which actually the whole of thinking[2290.95] [2290.95][S02]that I described is one of.[2292.7] [2292.7][S02]If we take every news article, but then we[2296.43] [2296.43][S02]want to create summaries, find how they relate to each other,[2299.17] [2299.17][S02]criticize some of them, this starts[2302.01] [2302.01][S02]to combine working memory, meaning[2304.68] [2304.68][S02]I'm going to have a scratchpad of the summaries, the issues[2308.67] [2308.67][S02]that I think I'm finding.[2310.11] [2310.11][S02]And that, when we call short or long context,[2312.96] [2312.96][S02]generally, we mean this last bit, the working memory,[2316.36] [2316.36][S02]whether you have a thousand tokens, which means[2319.29] [2319.29][S02]I couldn't possibly do much.[2321.04] [2321.04][S02]I can retrieve articles.[2322.78] [2322.78][S02]It's already over thousand words.[2324.37] [2324.37][S02]Not much I can do to summarize them.[2326.71] [2326.71][S02]Or it can be massive, in which case[2329.22] [2329.22][S02]you have many more possibilities to do reasoning on top of that,[2334.06] [2334.06][S02]and so on and so forth.[2335.02] [2335.02][S02]And so one of the breakthroughs of the year,[2338.47] [2338.47][S02]actually, we're still in 2024.[2341.38] [2341.38][S01] Just.[2342.25] [2342.25][S02] Yes, but it was just[2343.708] [2343.708][S02]to enable millions of tokens in context,[2345.85] [2345.85][S02]which enables many things.[2348.64] [2348.64][S02]You can retrieve something from the past,[2351.25] [2351.25][S02]but then bring it forward and then do[2352.88] [2352.88][S02]a very detailed analysis.[2354.57] [2354.57][S02]That's a bit of the examples of, we have a movie,[2357.56] [2357.56][S02]we can upload a movie or some very long video[2360.53] [2360.53][S02]and start doing summarization.[2362.37] [2362.37][S02]The fact we upload it is more episodic memory,[2365.4] [2365.4][S02]but then now we have it in memory.[2367.17] [2367.17][S02]It all fits in memory.[2368.22] [2368.22][S02]We can do quite a lot of associations within each frame,[2372.89] [2372.89][S02]each object in the movie, and so on and so forth.[2375.92] [2375.92][S01] Is a longer context window better always?[2379.17] [2379.17][S01]Because I'm just thinking about, I[2380.685] [2380.685][S01]don't know how much you guys are still using neuroscience[2383.06] [2383.06][S01]as an inspiration for what you're doing.[2384.727] [2384.727][S01]But the human memory, there's a limit to the working memory.[2388.582] [2388.582][S01]There's certainly some times where you're like,[2390.54] [2390.54][S01]my brain is full and I'm done.[2392.52] [2392.52][S02] Sometimes brain is an inspiration,[2395.22] [2395.22][S02]but computers certainly have advantages.[2397.26] [2397.26][S02]We should build on its strengths, right?[2399.36] [2399.36][S02]So perhaps the fact that they can have literally in memory[2403.944] [2403.944][S02]every Wikipedia article, whatever it is,[2407.88] [2407.88][S02]we can't, but if the model can, well, there you go.[2411.6] [2411.6][S02]You have new capabilities.[2412.8] [2412.8][S02]But also, it might be too confusing to have[2415.31] [2415.31][S02]too much information even for these neural networks.[2418.89] [2418.89][S02]So it might be a good idea to compress.[2421.44] [2421.44][S02]So that's where you probably want[2423.14] [2423.14][S02]to push for getting some inspiration for how we might[2426.86] [2426.86][S02]do what we do, which is quite amazing,[2428.45] [2428.45][S02]in terms of memory retrieval and so on.[2430.382] [2430.382][S01] Yeah.[2431.09] [2431.09][S01]This is why you're leading up Drastic Research.[2433.33] [2433.33][S02] Yes.[2434.88] [2434.88][S02]What we want to do with the models[2436.82] [2436.82][S02]should be definitely inspiring and forward looking.[2440.61] [2440.61][S02]And then what are the main limits of the technology?[2445.44] [2445.44][S02]And then try to, of course, place the bets[2447.59] [2447.59][S02]and inspire the teams to finding solutions[2449.84] [2449.84][S02]around the critical components.[2451.755] [2451.755][S01] But some of the bets that you've already made[2454.13] [2454.13][S01]have come off.[2454.92] [2454.92][S01]I know there's been a big announcement[2456.98] [2456.98][S01]of a dizzying number of new features[2461.36] [2461.36][S01]that have just come out.[2462.36] [2462.36][S01]Can we talk through some of them?[2463.97] [2463.97][S01]But then maybe also, talk to me about the different skills[2466.935] [2466.935][S01]that we've already spoken about and how[2468.56] [2468.56][S01]they appear in each of these.[2470.51] [2470.51][S02] Yeah, so we have quite a few systems[2476.77] [2476.77][S02]around our best Gemini models.[2479.6] [2479.6][S02]So one of the things that we've done is update to 2.0.[2483.71] [2483.71][S02]We're seeing a generational leap.[2485.42] [2485.42][S02]Even if you say, look, let's not scale anymore,[2487.73] [2487.73][S02]can we get better quality?[2488.96] [2488.96][S02]So we've done it sort of again.[2490.97] [2490.97][S02]These models are faster, they're cheaper,[2493.45] [2493.45][S02]and they're actually better.[2494.695] [2494.695][S01] Basically, Gemini has got better.[2496.57] [2496.57][S01]That's it.[2496.93] [2496.93][S02] Yeah, Gemini has got better, but not[2498.43] [2498.43][S02]only because we scaled.[2499.61] [2499.61][S01] Sure.[2500.318] [2500.318][S02] That's one of the main messages.[2502.58] [2502.58][S01] Tell me more about the agentic capabilities[2505.17] [2505.17][S01]that you've brought to Gemini.[2506.42] [2506.42][S02] Yeah, so we're releasing a companion in Chrome[2510.22] [2510.22][S02]where you can just type to do a task[2512.95] [2512.95][S02]that maybe some of these tasks are tricky.[2517.0] [2517.0][S02]Because I partly enjoy them, but also I partly don't like them.[2519.77] [2519.77][S02]So I'm thinking now very clearly about trips.[2522.136] [2522.136][S02]So you travel and you look for hotels or flights or whatnot,[2526.88] [2526.88][S02]and a lot of it feels like, oh, I wish this could be automated.[2531.13] [2531.13][S02]But at the same time, I wouldn't just[2533.34] [2533.34][S02]want to not be part of this journey.[2536.172] [2536.172][S02]So I guess the kind of thing we're releasing[2539.64] [2539.64][S02]is something that hopefully will automate parts of the more[2545.67] [2545.67][S02]trivial steps or repeated or things[2548.22] [2548.22][S02]that need automation because I can't be[2550.532] [2550.532][S02]bothered to click everything.[2551.74] [2551.74][S02]So we're adding an agent that you can ask it to do something[2557.64] [2557.64][S02]for you, and then it's going to, again, through thinking[2562.02] [2562.02][S02]and through acting on the basic clicking on links[2567.24] [2567.24][S02]and so on, try to solve the task for you.[2570.04] [2570.04][S02]And that's quite an exciting, both research challenge,[2574.47] [2574.47][S02]opportunity.[2575.1] [2575.1][S02]Because it's a very general environment for a very general[2578.01] [2578.01][S02]agent and model, ultimately.[2580.18] [2580.18][S02]And some examples that we've had again in the early prototypes,[2583.69] [2583.69][S02]we can ask to play a game, which of course, goes back[2586.38] [2586.38][S02]to the roots of DeepMind, on the browser.[2590.55] [2590.55][S02]And it did OK.[2592.14] [2592.14][S02]It finds a website, it starts playing the game.[2594.69] [2594.69][S02]It's kind of a cool connection to the more general you are,[2598.41] [2598.41][S02]then the more you can treat environments[2601.88] [2601.88][S02]where you had specialization in the past, as now,[2605.1] [2605.1][S02]oh, I just can type it and it just[2606.86] [2606.86][S02]goes and learn to play this game.[2608.79] [2608.79][S02]We're not quite there, but this is[2610.67] [2610.67][S02]a glimpse of maybe where we could go[2612.74] [2612.74][S02]with this kind of technology.[2615.0] [2615.0][S01] You're right that it does really[2616.88] [2616.88][S01]bring us back to that thing that you were doing[2619.52] [2619.52][S01]so many years ago, which was something that could[2621.62] [2621.62][S01]use a keyboard and a mouse.[2623.54] [2623.54][S01]It's a really similar thing.[2625.02] [2625.02][S02] Yeah, even the actions are very similar.[2627.312] [2627.312][S02]They understand the screen and, given[2629.6] [2629.6][S02]what you ask, where would you click and so on,[2632.13] [2632.13][S02]that's the same sort of actions even that very general games[2638.045] [2638.045][S02]you would have to interact with.[2639.47] [2639.47][S02]The difference is the goal there is narrow.[2642.63] [2642.63][S02]It's just one game and the same kind of screens.[2644.88] [2644.88][S02]Whereas here, it's the whole web, which is pretty vast.[2647.595] [2647.595][S01] Well, OK.[2648.47] [2648.47][S01]But then so I'm sort of imagining what you could do now.[2651.048] [2651.048][S01]Could it look in your calendar?[2652.34] [2652.34][S01]Could you say, I want to go on holiday next year,[2654.41] [2654.41][S01]and it could look in your calendar[2655.827] [2655.827][S01]and work out when the best week was,[2657.5] [2657.5][S01]know your budget, et cetera, et cetera, et cetera?[2660.44] [2660.44][S02] Yeah, so these models[2662.65] [2662.65][S02]are not far from being able to automate this.[2664.7] [2664.7][S02]So now it's a matter of making it better, making it safe.[2668.66] [2668.66][S02]There's a lot of steps.[2669.89] [2669.89][S02]But if you just fast forward, anything[2673.81] [2673.81][S02]a human can do on a browser, these things[2676.54] [2676.54][S02]can do in principle.[2678.35] [2678.35][S02]And then if you make them really understand[2680.35] [2680.35][S02]what you want and really good through thinking[2683.08] [2683.08][S02]and other techniques, they'll get better and better.[2685.87] [2685.87][S02]And they'll be probably faster and maybe in some cases[2689.38] [2689.38][S02]much better than you at doing those.[2691.25] [2691.25][S02]So that's the dream.[2693.4] [2693.4][S02]And these are super early stages,[2696.02] [2696.02][S02]but it's also super exciting.[2697.85] [2697.85][S02]And I think certainly next year, we're[2700.51] [2700.51][S02]going to see a lot of experimentation[2702.79] [2702.79][S02]around this idea of intersecting language models agentically[2707.52] [2707.52][S02]with browser or computers more generally.[2710.74] [2710.74][S01] How about coding?[2712.48] [2712.48][S02] Yeah, coding is a great one as well.[2714.88] [2714.88][S02]We are also releasing tools for software engineering, which[2718.86] [2718.86][S02]of course, they generally require not only, hey, here[2722.94] [2722.94][S02]is a perfect description of a puzzle about coding[2725.91] [2725.91][S02]and please write me the code.[2727.33] [2727.33][S02]And by the way, I know how to test it.[2729.19] [2729.19][S02]It's more iterative.[2730.23] [2730.23][S02]You have to write code and run the code and so on and so forth.[2734.5] [2734.5][S02]So we are putting that capability, as well, forward[2737.91] [2737.91][S02]from an agentic point of view.[2742.21] [2742.21][S02]Games are very important.[2743.85] [2743.85][S02]And of course, that was a means to an end[2746.13] [2746.13][S02]to develop powerful algorithms.[2747.94] [2747.94][S02]But it's also very interesting to think[2749.73] [2749.73][S02]about how these very powerful multimodal[2753.72] [2753.72][S02]models start to understand games and can aid users to entertain[2758.7] [2758.7][S02]doing a game session, give them advice, or tell[2764.19] [2764.19][S02]a joke about the game, or whatnot.[2765.85] [2765.85][S02]So we're also experimenting with this sort of game companion.[2770.622] [2770.622][S01] OK, all of these things[2772.08] [2772.08][S01]that you're talking about, this is[2773.52] [2773.52][S01]sounding very close to intelligence[2777.46] [2777.46][S01]that is quite general.[2778.96] [2778.96][S01]Are we getting close to AGI?[2780.46] [2780.46][S02] Yeah, this is a good question.[2782.335] [2782.335][S02]Look, I was thinking about this earlier this week.[2786.93] [2786.93][S02]If 10 years ago, 5 years ago even, I[2789.9] [2789.9][S02]would have been given the models today,[2792.34] [2792.34][S02]and I would say, look, there's a secret lab, this is a model,[2797.19] [2797.19][S02]play with it and tell me if you think this is actually[2800.58] [2800.58][S02]close to a general intelligence; I would have claimed,[2803.62] [2803.62][S02]oh, yeah, that comes from a future where AGI basically[2807.33] [2807.33][S02]either has happened or I can see that this is very close to it.[2811.08] [2811.08][S02]So the closer you are, the more you find,[2814.84] [2814.84][S02]oh, but it hallucinates.[2815.885] [2815.885][S02]Of course, that's very important.[2817.26] [2817.26][S02]But I think, just zooming out, it just[2820.35] [2820.35][S02]feels like, OK, it's getting pretty close.[2823.357] [2823.357][S01] But then DeepMind's mission statement,[2825.44] [2825.44][S01]solve intelligence, that sort of intelligence,[2828.36] [2828.36][S01]super intelligence, something that[2832.37] [2832.37][S01]surpasses human intelligence, do you think that scaling[2835.7] [2835.7][S01]is enough to get us there?[2837.18] [2837.18][S01]Or do you think that we need something else?[2839.67] [2839.67][S02] Yeah, Google DeepMind[2842.0] [2842.0][S02]has this mission to obviously intersect intelligence[2844.82] [2844.82][S02]with science to push the boundaries.[2846.618] [2846.618][S02]And we've seen a good example very recently, of course,[2848.91] [2848.91][S02]with AlphaFold.[2850.2] [2850.2][S02]So in that sense, from a domains perspective,[2855.41] [2855.41][S02]we honestly have seen some examples[2857.57] [2857.57][S02]already of narrow but super intelligent system.[2861.03] [2861.03][S02]AlphaFold was only doing that.[2862.95] [2862.95][S02]And I think probably that's the domains[2868.04] [2868.04][S02]to think about where we're going to start[2869.84] [2869.84][S02]seeing super intelligence, even from the general sort[2873.47] [2873.47][S02]of capabilities these models have.[2876.15] [2876.15][S02]You might need to do some specialization.[2878.07] [2878.07][S02]And again, it might be worth it.[2879.63] [2879.63][S02]Was it worth it to solve protein folding?[2882.29] [2882.29][S02]Absolutely, right?[2883.96] [2883.96][S02]But I think that's a good test to use.[2888.56] [2888.56][S02]And we are very well positioned.[2890.17] [2890.17][S02]Because we, of course, have the whole science team[2892.94] [2892.94][S02]and so on working on very interesting problems.[2896.74] [2896.74][S02]Now, if you take the language models[2901.87] [2901.87][S02]and you start thinking about agents,[2904.67] [2904.67][S02]putting them in environments that[2906.16] [2906.16][S02]could be more about science, simulation, theorem provers,[2910.75] [2910.75][S02]and so on, will something very discrete[2913.96] [2913.96][S02]be needed to enable other breakthroughs?[2916.67] [2916.67][S02]I would say probably not.[2918.31] [2918.31][S02]Without another transformer-like breakthrough,[2921.14] [2921.14][S02]perhaps, feels like we're going to start[2923.5] [2923.5][S02]seeing more examples of, oh my god, yeah, like in math now,[2927.64] [2927.64][S02]it just discovers new theorems that mathematicians[2930.85] [2930.85][S02]find interesting.[2931.99] [2931.99][S02]And it happened by just, of course, very good execution[2936.74] [2936.74][S02]plus scaling up of some of the ideas and so on and so forth.[2940.53] [2940.53][S01] It is interesting, though, that the first dominoes[2943.6] [2943.6][S01]to fall are the ones which have a ground truth--[2945.79] [2945.79][S02] Yeah.[2946.18] [2946.18][S01] --like science, as you describe.[2948.2] [2948.2][S02] Yeah.[2949.033] [2949.033][S02]Although, yeah, science, it depends which sciences might[2952.24] [2952.24][S02]have ground truth, I suppose.[2953.6] [2953.6][S01] Protein folding, definitely.[2955.267] [2955.267][S02] Yeah, it's true.[2957.28] [2957.28][S02]I'm hoping we also see some other ways[2960.73] [2960.73][S02]to advance in a superhuman way.[2963.06] [2963.06][S02]You could imagine having a brainstorming scientific advisor[2967.96] [2967.96][S02]that is powered by one of these powerful models.[2970.51] [2970.51][S02]And more than it discovers something[2972.7] [2972.7][S02]or it proves something new, it just challenges your assumptions[2975.86] [2975.86][S02]and it makes you think out of the box in a way[2978.25] [2978.25][S02]that, then, my creativity sort of gets me[2981.22] [2981.22][S02]to a place I couldn't have gone.[2982.82] [2982.82][S02]Then you would call that superhuman in some ways as well.[2986.215] [2986.215][S02]So I think those are definitely not out of scope[2988.99] [2988.99][S02]and much harder to also, of course,[2990.89] [2990.89][S02]think of how do you reward that behavior.[2993.392] [2993.392][S01] Absolutely fascinating.[2994.85] [2994.85][S01]There was definitely a lot of drastic stuff in there.[2997.058] [2997.058][S02] Yeah.[2997.892] [2997.892][S02]Yes.[2998.53] [2998.53][S01] Thank you so much for joining me.[2999.91] [2999.91][S02] Yeah, likewise.[3000.69] [3000.69][S02]Thanks, pleasure.[3001.418] [3001.418][S02]See you in five years.[3002.335] [3002.335][S01] [LAUGHS] I think there was this real theme that[3005.63] [3005.63][S01]emerged from that conversation, at least for me anyway,[3007.95] [3007.95][S01]which was this idea of generality.[3011.22] [3011.22][S01]And if you think about it, there is this generality in the way[3015.56] [3015.56][S01]that intelligence advances knowledge.[3017.94] [3017.94][S01]Like those old astronomers like Copernicus,[3020.37] [3020.37][S01]they were assessing lots of data from observing the sky[3023.93] [3023.93][S01]and using that to extract a model of the solar system.[3028.94] [3028.94][S01]But in the case of AlphaGo, it was observing games of Go[3033.05] [3033.05][S01]to extract a model for the best possible way to play.[3037.83] [3037.83][S01]And now, somewhere embedded in everything[3040.7] [3040.7][S01]that has ever been created by humans[3042.2] [3042.2][S01]is this model, this underlying truth[3045.05] [3045.05][S01]of how we experience reality.[3047.87] [3047.87][S01]And the model that we're looking for, of course,[3050.22] [3050.22][S01]it's never going to be as neat as heliocentrism.[3053.0] [3053.0][S01]But that model does seem to be in there, hidden[3056.98] [3056.98][S01]among the frozen weights of Gemini.[3060.05] [3060.05][S01]Now, if that's what we've done so far,[3062.36] [3062.36][S01]the next phase is to try and use those general ideas[3065.5] [3065.5][S01]to extract a model of human preferences, too.[3069.29] [3069.29][S01]And that is, of course, a lot, lot harder.[3073.43] [3073.43][S01]But if we succeed, it might just get us to a more general form[3078.04] [3078.04][S01]of intelligence, too, AGI.[3081.94] [3081.94][S01]Now, if you found this conversation interesting,[3084.11] [3084.11][S01]I think it's worth also checking out[3085.66] [3085.66][S01]the episodes I did with Jeff Dean[3087.34] [3087.34][S01]on, among other things, scaling, and Iason Gabriel[3091.18] [3091.18][S01]on the ethics of AI agents.[3093.44] [3093.44][S01]Or if you want to dig a bit deeper into the development[3096.76] [3096.76][S01]of Gemini 2.0, then you can check out the latest episode[3100.69] [3100.69][S01]of the new "Google AI Release Notes"[3103.09] [3103.09][S01]podcast with host Logan Kilpatrick.[3106.4] [3106.4][S01]This and other episodes can be found[3108.58] [3108.58][S01]wherever you get your podcasts.[3110.72] [3110.72][S01]Until next time.[3112.54] [3112.54][MUSIC PLAYING][3116.19]