[0.0][MUSIC PLAYING][3.458] [6.916][S01] Welcome back to "Google DeepMind: The Podcast"[9.76] [9.76][S01]with me, your host, Professor Hannah Fry.[12.13] [12.13][S01]In this episode, we get to speak to one[14.35] [14.35][S01]of the most legendary figures in the world of computer[17.23] [17.23][S01]science, Jeff Dean.[19.2] [19.2][S01]He was there in the late 1990s writing the code that would turn[22.81] [22.81][S01]Google from a small startup to the multinational company it is[27.52] [27.52][S01]today.[28.4] [28.4][S01]Jeff spearheaded TensorFlow, one of the programming tools[31.75] [31.75][S01]responsible for the democratization[33.67] [33.67][S01]of machine learning.[35.06] [35.06][S01]He's the one who pushed the boundaries[36.97] [36.97][S01]of artificial intelligence in the direction[39.22] [39.22][S01]of large-scale models.[41.24] [41.24][S01]And if that wasn't enough, he also co-founded[44.29] [44.29][S01]Google's AI research project, Google Brain,[47.42] [47.42][S01]and was one of the earliest pioneers of a new neural network[50.86] [50.86][S01]architecture called Transformers.[53.36] [53.36][S01]Not sure that one's going to catch on.[55.27] [55.27][S01]I think this is why people joke that Jeff Dean's resume just[58.84] [58.84][S01]lists the things that he hasn't done.[60.39] [60.39][S01]It's shorter that way.[61.9] [61.9][S01]More recently, Jeff, as Google's chief scientist,[64.819] [64.819][S01]has occupied one of the most important seats[66.89] [66.89][S01]around the table as the two great AI arms of Alphabet[70.31] [70.31][S01]have merged, DeepMind and Google Brain.[73.2] [73.2][S01]His latest baby, which he co-parents,[75.39] [75.39][S01]is known as Gemini, which takes large language models well[78.89] [78.89][S01]beyond language alone.[80.67] [80.67][S01]Gemini is a multimodal model which can understand text, code,[85.05] [85.05][S01]audio, images, and video.[87.36] [87.36][S01]It is AI through and through and almost certainly the direction[91.79] [91.79][S01]that Google Search itself is heading in.[94.265] [94.265][S01]Jeff, thank you so much for joining me.[95.89] [95.89][S02] Thank you for having me.[97.08] [97.08][S02]It's a delight to be here.[98.24] [98.24][S01] So, OK, 25 years, a quarter of a century at Google I[102.38] [102.38][S01]want to a little bit of what it was like back in the early days,[105.055] [105.055][S01]right, like in the '90s, when you first joined,[107.22] [107.22][S01]when Google wasn't the sort of slick organization that it is[109.9] [109.9][S01]now.[110.4] [110.4][S01]Was it all a lot of laptops with stickers on it and sort of[113.645] [113.645][S01]coding in flip-flops?[115.25] [115.25][S02] Sadly, it was pre-laptops.[116.87] [116.87][S01] Pre-laptops.[117.87] [117.87][S02] Well, mostly.[119.03] [119.03][S02]Yeah, we all had those giant CRT-based--[120.71] [120.71][S01] Yeah, of course.[121.25] [121.25][S02] --monitors, those pre-LCD monitors.[123.63] [123.63][S02]So they took up a lot of desk space.[126.02] [126.02][S01] Not very portable.[127.27] [127.27][S02] My desk was like a door on two sawhorses.[131.21] [131.21][S02]That was the-- and you could adjust it yourself[134.36] [134.36][S02]by getting under the desk and standing up with your back[136.82] [136.82][S02]to get it to the next higher setting.[138.86] [138.86][S01] Really?[139.76] [139.76][S02] Yeah.[141.057] [141.057][S01] Amazing.[141.89] [141.89][S02] And when I started, we[143.265] [143.265][S02]were in this small office area, actually not-- maybe three times[146.33] [146.33][S02]as big as this room.[148.117] [148.117][S01] The whole of Google.[149.45] [149.45][S02] The whole of Google on University Avenue[151.575] [151.575][S02]in Palo Alto above what's now a T-Mobile cell phone store.[156.2] [156.2][S02]And the really fun and exciting thing in those days[162.11] [162.11][S02]was we were a small company, but we[166.61] [166.61][S02]could see that people were using our service more and more[169.13] [169.13][S02]because we were providing good-quality, high-quality[172.28] [172.28][S02]search.[173.42] [173.42][S02]And you could see your traffic growing[175.82] [175.82][S02]day over day, week over week.[177.65] [177.65][S02]So we'd always be trying to not melt on[179.9] [179.9][S02]Tuesday at noon, which was the peak traffic hour of the week.[184.14] [184.14][S02]And that would require we deploy more computers quickly,[188.3] [188.3][S02]that we optimize our code to make it run faster,[191.37] [191.37][S02]that we come up with new and interesting innovations[194.45] [194.45][S02]for next month's index that make it able to serve more[197.9] [197.9][S02]users with the same hardware.[199.108] [199.108][S01] I can imagine it being very exciting.[201.15] [201.15][S01]Was there a moment when you guys realized--[203.03] [203.03][S01]was there a moment when you were like,[204.613] [204.613][S01]ah, OK, this is really going to be big?[206.63] [206.63][S02] I mean, I think you could[208.13] [208.13][S02]see that from the very earliest days when I was joining--[211.11] [211.11][S02]I joined the company because our traffic was growing really fast,[214.49] [214.49][S02]and we felt like by focusing on returning really high-quality[220.25] [220.25][S02]search results and doing that quickly and giving users what[224.66] [224.66][S02]they want--[225.3] [225.3][S02]we actually want to get people off our site[227.63] [227.63][S02]as quickly as possible to the information they need--[230.96] [230.96][S02]that was a winning proposition.[233.73] [233.73][S02]And users seemed to like our service.[235.83] [235.83][S02]And so it seemed reasonably promising,[240.21] [240.21][S02]I would say, from even the early days.[241.793] [241.793][S01] There's quite a big gap[243.252] [243.252][S01]between reasonably promising, though,[244.81] [244.81][S01]and sort of what it ended up being.[246.6] [246.6][S01]Has it been a surprise to you-- like, to all of you?[249.74] [249.74][S02] I mean, I think there's been a bunch of things[252.14] [252.14][S02]that we've branched out into that were obviously[255.308] [255.308][S02]hard to anticipate.[256.1] [256.1][S02]Autonomous vehicles-- it's hard to fathom[259.04] [259.04][S02]that when you're working on a search engine.[261.95] [261.95][S02]But I think the sort of gradual broadening out[265.67] [265.67][S02]of our portfolio of products to other kinds of information[269.54] [269.54][S02]makes a lot of sense.[270.72] [270.72][S02]So going from public web pages to helping users organize[275.27] [275.27][S02]their own email with Gmail, things[277.31] [277.31][S02]like that, those are sort of natural evolutions of things[280.58] [280.58][S02]that solve real problems people have.[284.48] [284.48][S02]And they get you in the state of, OK, well,[287.1] [287.1][S02]now we don't just have one product.[289.53] [289.53][S02]We have a handful of products that people[292.22] [292.22][S02]use fairly regularly.[293.345] [293.345][S01] And I sort of wonder.[294.72] [294.72][S01]Looking back through all that time,[296.178] [296.178][S01]do you think that Google was always a search company,[299.57] [299.57][S01]or do you think it was an AI company sort of pretending[302.48] [302.48][S01]to be a search company?[303.807] [303.807][S02] Yeah, I mean, I think a lot[305.39] [305.39][S02]of the problems we wanted to tackle as a company really[309.23] [309.23][S02]were sort of ones that would require[312.53] [312.53][S02]AI to really, truly solve.[314.22] [314.22][S02]And so along the way, in a long period of time, 25 years,[319.01] [319.01][S02]we've been progressively tackling some of those hard AI[323.09] [323.09][S02]problems and making progress on them[324.62] [324.62][S02]and then using the new techniques that are now[327.32] [327.32][S02]starting to work in the midst of search[329.9] [329.9][S02]and in the midst of all of our other products.[332.04] [332.04][S01] Do you think that Google will always[334.04] [334.04][S01]be a search company, or do you think-- is it[336.08] [336.08][S01]even a search company now?[337.32] [337.32][S01]Is it changing?[338.54] [338.54][S02] Well, one of the things[339.957] [339.957][S02]I really like about Google is our mission[341.93] [341.93][S02]is still incredibly relevant even 25 years later-- organize[348.022] [348.022][S02]the world's information and make it universally[349.98] [349.98][S02]accessible and useful.[351.66] [351.66][S02]And I feel like Gemini is really helping[354.18] [354.18][S02]us push in the direction of understanding[357.33] [357.33][S02]lots of different kinds of information--[359.53] [359.53][S02]so textual data, software code, which is kind of texty in nature[365.83] [365.83][S02]but very structured in certain ways,[368.08] [368.08][S02]but also all the other kinds of modalities of input[371.46] [371.46][S02]that humans are sort of fluent in.[373.57] [373.57][S02]We're naturally-- we read stuff, but we also[376.56] [376.56][S02]see stuff with our eyes and hear stuff with our ears.[379.95] [379.95][S02]And you want models to be able to sort of take in information[383.91] [383.91][S02]in its many forms and also produce information in text form[390.09] [390.09][S02]or maybe generate audio so that you[392.22] [392.22][S02]can have a conversation with the model[395.04] [395.04][S02]or produce imagery, if that's appropriate,[397.68] [397.68][S02]or annotate text with graphs or things like that.[400.84] [400.84][S02]So we're really trying to make a single model that[403.95] [403.95][S02]can take in all those modalities and produce all those modalities[406.77] [406.77][S02]and use that capability when it makes sense.[410.718] [410.718][S01] Can you remember when you first[412.51] [412.51][S01]came across neural networks?[414.41] [414.41][S02] Oh, yeah, actually.[416.35] [416.35][S02]So neural networks have had an interesting history.[420.044] [420.044][S02]AI is a quite old discipline, and the early phases of AI[424.78] [424.78][S02]were about how do we define rules about how things work.[429.11] [429.11][S02]So that was like the '50s, '60s, '70s, to some extent.[433.39] [433.39][S02]And then neural networks kind of came along in the '70s and had[438.61] [438.61][S02]a wave of excitement in the late '80s and early '90s.[442.36] [442.36][S02]And I was actually an undergrad in 1990[445.39] [445.39][S02]at University of Minnesota.[446.81] [446.81][S02]And I was taking a class in parallel processing, which[449.44] [449.44][S02]is this idea of how do you take problems and break them down[453.49] [453.49][S02]into pieces that can be done on different computers.[455.78] [455.78][S02]And then kind of in conjunction, all those computers[459.97] [459.97][S02]work together to solve a single problem.[462.305] [462.305][S01] I guess this is the point where the computing[464.68] [464.68][S01]power wasn't quite what we have today.[466.88] [466.88][S01]So how do you make computers work as a team?[468.8] [468.8][S02] Right, yeah.[469.758] [469.758][S02]And at that time, neural networks[472.49] [472.49][S02]was a sort of particular approach to machine learning[476.93] [476.93][S02]and AI that involved kind of very kind[481.79] [481.79][S02]of crude approximations of how we[484.49] [484.49][S02]think real human or other brains work with neurons.[487.57] [487.57][S02]So that's why they're called neural networks because they're[490.07] [490.07][S02]made up of artificial neurons.[492.74] [492.74][S02]And artificial neurons have connections to other neurons[496.04] [496.04][S02]below them.[496.89] [496.89][S02]And then they look at kind of the signals that come up[500.18] [500.18][S02]from those artificial neurons, and they[502.07] [502.07][S02]decide how interested are they in that particular pattern[506.75] [506.75][S02]of signals.[507.57] [507.57][S02]And then they decide, should they[509.66] [509.66][S02]be excited enough to send a signal further[512.87] [512.87][S02]up the neural network?[514.32] [514.32][S02]And so that's one artificial neuron.[516.09] [516.09][S02]And a neural network is made up of lots of layers[519.74] [519.74][S02]of lots of these neurons.[521.64] [521.64][S02]And so the higher layer neurons build on the representations[525.65] [525.65][S02]of the lower neurons.[527.3] [527.3][S02]And so if you're, for example, building a neural network[530.66] [530.66][S02]for image problems, the lowest neurons in the lowest layer[535.46] [535.46][S02]might learn features like, oh, it's a splotch of red or green,[540.0] [540.0][S02]or there's an edge at this orientation.[542.04] [542.04][S02]And then the next level up might learn, oh,[544.55] [544.55][S02]it's an edge with yellow on one side.[548.6] [548.6][S02]And then higher up, it might be, oh, it looks[551.51] [551.51][S02]like a nose or ears or a face.[555.17] [555.17][S02]And so by building these layered, learned abstractions,[560.09] [560.09][S02]these systems can actually develop[562.01] [562.01][S02]very powerful pattern-recognition[564.71] [564.71][S02]capabilities.[565.29] [565.29][S02]And that's what people were excited[566.748] [566.748][S02]about about neural networks in kind of 1985, 1990.[570.075] [570.075][S01] But we're talking teeny, teeny, tiny little--[572.45] [572.45][S02] Teeny networks, yeah.[573.783] [573.783][S02]So they could not recognize faces and cars and things.[578.96] [578.96][S02]They could recognize little patterns in artificial--[584.07] [584.07][S02]artificially generated patterns.[585.75] [585.75][S01] Yeah.[585.95] [585.95][S01]Like, you have a grid, and it can recognize maybe[587.992] [587.992][S01]a cross or something.[588.89] [588.89][S02] Or a handwritten digit.[590.307] [590.307][S02]Is it a seven or an eight?[591.518] [591.518][S01] That's fancy.[592.56] [592.56][S02] Yeah, that was fancy.[593.893] [593.893][S02]But that was kind of what they could do at that time.[596.6] [596.6][S02]But people were excited because they[598.37] [598.37][S02]could solve those kinds of problems[600.26] [600.26][S02]that other systems, based on purely logically specified[604.07] [604.07][S02]rules of what a seven means, weren't actually[607.25] [607.25][S02]able to do very well in a way that[609.17] [609.17][S02]generalized to all kinds of messy, handwritten sevens.[612.39] [612.39][S02]So I was kind of intrigued by that after my two lectures[615.86] [615.86][S02]on neural nets, and I decided I would[618.56] [618.56][S02]do a senior thesis, honors thesis,[622.52] [622.52][S02]on parallel training of neural networks[624.27] [624.27][S02]because I felt like we just need more compute.[626.73] [626.73][S02]What if we use the 32 processor machine in the department[630.14] [630.14][S02]and made a bigger system that we could train bigger neural nets?[634.58] [634.58][S02]So that was what I spent a couple of months--[637.43] [637.43][S02]three months on.[638.39] [638.39][S01] Did it work?[639.09] [639.09][S02] Yeah, yeah.[640.007] [640.007][S02]So anyway, I was very excited.[641.727] [641.727][S02]I was like, oh, 32 processors is going[643.31] [643.31][S02]to cause neural nets to really, really hum--[645.325] [645.325][S01] Take off.[646.2] [646.2][S02] --and sing.[647.48] [647.48][S02]Turns out I was wrong, naive undergrad me.[651.23] [651.23][S02]We needed about a million times as much processing power[653.99] [653.99][S02]to get them to really start to work well on real problems[656.78] [656.78][S02]that you might sort of care about.[658.382] [658.382][S01] Yeah.[659.09] [659.09][S02] But then, thanks to 20 years[661.43] [661.43][S02]of progress of Moore's law and much faster CPUs[665.45] [665.45][S02]and computational devices and stuff,[667.56] [667.56][S02]we actually then started to have practical systems that[671.48] [671.48][S02]had a million times as much compute as even[673.52] [673.52][S02]our fancy 32 processor machine.[676.68] [676.68][S02]And so I started to get interested in neural nets[680.51] [680.51][S02]again when Andrew Ng, who's a Stanford faculty member,[684.96] [684.96][S02]was consulting at Google one day a week.[686.79] [686.79][S02]And I bumped into him in one of our many micro kitchens.[689.918] [689.918][S02]I'm like, oh, what are you doing at Google?[691.71] [691.71][S02]He's like, well, I haven't really[693.085] [693.085][S02]figured it out yet because I just started consulting here.[695.79] [695.79][S02]But some of my students at Stanford[697.73] [697.73][S02]are getting good results on neural networks.[699.92] [699.92][S02]I'm like, oh, really?[701.22] [701.22][S02]Why don't we train really, really big neural networks?[703.56] [703.56][S02]So that was the genesis of our work[705.71] [705.71][S02]on neural networks at Google.[707.19] [707.19][S02]And then we formed a small team called the Google Brain[710.18] [710.18][S02]Team to start looking at how could we[713.21] [713.21][S02]train very large neural networks using Google's[716.06] [716.06][S02]computational resources.[717.74] [717.74][S02]And so we sort of built this software infrastructure that[723.14] [723.14][S02]enabled us to take a neural network description and then[726.5] [726.5][S02]break it down into pieces that would be done on different[730.13] [730.13][S02]computers, different members of this parallel team,[733.88] [733.88][S02]and then communicate amongst themselves in ways that they[737.51] [737.51][S02]needed to do in order to sort of tackle the overall problem[740.9] [740.9][S02]of how do you train a single neural network on 2,000[744.2] [744.2][S02]computers.[745.58] [745.58][S02]And so that was kind of the earliest software[747.86] [747.86][S02]we built for really scaling up neural network training.[751.49] [751.49][S02]And it enabled us to train models that were 50 to 100 times[755.33] [755.33][S02]larger than sort of existing neural networks.[758.2] [758.2][S01] 2011, right?[759.64] [759.64][S02] Yeah, this was like early 2012.[762.39] [762.39][S01] So this is before the big breakthroughs[765.86] [765.86][S01]in image recognition.[767.2] [767.2][S01]This is like way back.[768.53] [768.53][S01]And in many ways, you were doing then the same thing[771.08] [771.08][S01]that you were doing previously of just[772.97] [772.97][S01]kind of stitching computers together.[775.292] [775.292][S02] Just like my undergrad thesis.[777.0] [777.0][S01] Exactly.[777.295] [777.295][S01]But again--[777.8] [777.8][S02] I'm like, hey, we could do that[778.76] [778.76][S02]again but at big scale.[780.227] [780.227][S01] And yet, this time--[781.56] [781.56][S02] This time, it actually[782.48] [782.48][S02]worked because the computers were faster,[783.89] [783.89][S02]and we used a lot more of them.[785.14] [785.14][S01] Did it feel like a bit of a gamble,[787.098] [787.098][S01]though, back in 2011?[788.36] [788.36][S02] Oh, yeah.[789.52] [789.52][S02]The system that we built for training these neural networks[793.34] [793.34][S02]and trying different ways of breaking them apart,[795.71] [795.71][S02]I actually named it DistBelief because it's--[799.98] [799.98][S02]partly because people didn't think it was really[801.98] [801.98][S02]going to work and also because it was a distributed[805.58] [805.58][S02]system that could build these--[808.01] [808.01][S02]one of the things we wanted to train[809.51] [809.51][S02]was belief networks in addition to neural networks.[811.68] [811.68][S01] Oh, I love it.[812.27] [812.27][S01]I love it.[812.88] [812.88][S02] So DistBelief it was.[814.22] [814.22][S01] DistBelief.[815.178] [815.178][S01]Amazing.[815.7] [815.7][S01]So, while this was going on stateside,[818.82] [818.82][S01]this side of the Atlantic, it was the beginnings of DeepMind.[822.475] [822.475][S02] Yes.[823.1] [823.1][S01] And I know that you were[825.62] [825.62][S01]the person tasked with coming over and checking them out,[828.605] [828.605][S01]right?[829.105] [829.105][S01]Can you tell me about that story?[830.61] [830.61][S02] Yeah, so actually, Geoffrey Hinton, who's[835.04] [835.04][S02]a very well-known machine learning researcher,[837.51] [837.51][S02]spent a summer at Google in 2011, I think,[842.12] [842.12][S02]and we couldn't figure out how to classify him.[844.56] [844.56][S02]So he got classified as an intern, which is a little funny.[847.603] [847.603][S01] Most senior intern in the entire history[849.77] [849.77][S01]of the world.[850.5] [850.5][S02] And so he and I were working together.[853.11] [853.11][S02]And then somehow, we found out about DeepMind.[855.405] [855.405][S02]I think Geoffrey had known a little bit[857.03] [857.03][S02]about sort of the formation of the company.[859.08] [859.08][S02]And some other people also said, oh, yeah, there's[861.402] [861.402][S02]this company over in the UK doing interesting--[863.36] [863.36][S01] And it was teeny tiny at this time.[864.67] [864.67][S01]I mean, like--[865.253] [865.253][S02] Yeah, like probably 40 or 50 people or something.[869.18] [869.18][S02]And so we decided as a company that we would go check them out[873.53] [873.53][S02]as a potential acquisition.[875.96] [875.96][S02]And so I was in California.[878.49] [878.49][S02]Geoffrey was back in Toronto, where he was[881.03] [881.03][S02]a faculty member at that time.[883.7] [883.7][S02]And Geoff has a bad back, so he can't actually fly commercially[888.62] [888.62][S02]because he can't sit down.[890.19] [890.19][S02]He can only lay down or stand up.[893.15] [893.15][S02]Airlines don't like it when you stand up during takeoff.[897.47] [897.47][S02]So we had to figure out a solution, which[900.05] [900.05][S02]was to get a medical bed in a private plane.[902.97] [902.97][S02]And so a bunch of us took off at California, flew to Toronto,[908.66] [908.66][S02]scooped Geoffrey up from the tarmac,[910.83] [910.83][S02]put him in the medical bed, and then we all flew to the UK[916.19] [916.19][S02]and landed in some--[917.9] [917.9][S02]not one of the main airports.[919.62] [919.62][S02]It was at the edge of town.[923.04] [923.04][S02]And we all got in a big van, and we trooped off[925.13] [925.13][S02]to visit DeepMind, which was near Russell Square, I think.[929.42] [929.42][S02]And we were really tired from flying the previous night.[933.96] [933.96][S02]But then we got 13 straight 20-minute lectures in a row[939.977] [939.977][S02]of all the different things they were doing.[941.81] [941.81][S01] What, from the team at DeepMind?[943.11] [943.11][S02] Yeah, from the team.[943.625] [943.625][S01] Oh wow.[944.31] [944.31][S02] So we have some work, some--[945.74] [945.74][S01] While jet lagged.[946.71] [946.71][S02] While jet lagged.[947.93] [947.93][S01] It's like something from a sitcom.[949.847] [949.847][S02] Yeah, yeah, exactly.[951.45] [951.45][S02]So then we got some presentations on some[956.08] [956.08][S02]of the Atari work they were doing,[958.01] [958.01][S02]which was published later, on how do you use reinforcement[962.71] [962.71][S02]learning to learn to play old Atari 2600 games--[966.38] [966.38][S02]so games like "Breakout" or "Pong" or various other ones--[970.87] [970.87][S02]which was quite interesting.[972.58] [972.58][S02]And then--[973.18] [973.18][S01] Because you guys hadn't[973.84] [973.84][S01]been doing the reinforcement learning at that time.[975.74] [975.74][S02] Right.[976.448] [976.448][S02]We'd mostly been focusing on how do[978.07] [978.07][S02]you scale up large-scale supervised and unsupervised[981.7] [981.7][S02]learning.[982.6] [982.6][S02]And--[983.1] [983.1][S01] And the reinforcement learning[984.85] [984.85][S01]is much more motivated by rewards.[986.9] [986.9][S02] Yeah, so I think all of these techniques[990.19] [990.19][S02]are really useful, and they're often useful in combination.[992.96] [992.96][S02]So reinforcement learning you should[994.54] [994.54][S02]think of as you have some agent operating in an environment,[1000.51] [1000.51][S02]and at every step, there's a bunch of different moves[1004.26] [1004.26][S02]you could make or actions you could take.[1006.57] [1006.57][S02]So in the game of "Go," for example,[1009.02] [1009.02][S02]you could play a stone in any of a whole bunch of different[1012.27] [1012.27][S02]positions.[1012.93] [1012.93][S02]In Atari, you could move your joystick up,[1015.54] [1015.54][S02]down, or left or right, or you could[1017.04] [1017.04][S02]push the left or right button.[1019.59] [1019.59][S02]And often, in these situations, you[1022.05] [1022.05][S02]don't get an immediate reward.[1024.06] [1024.06][S02]So in "Go," you play a stone, and you don't know[1026.31] [1026.31][S02]if that's a good idea or not until the whole process[1030.72] [1030.72][S02]of the rest of the game plays out.[1032.64] [1032.64][S02]And one of the interesting things about reinforcement[1035.069] [1035.069][S02]learning is it's able to take kind[1037.44] [1037.44][S02]of long sequences of actions and then[1040.319] [1040.319][S02]attribute rewards or negative rewards[1043.89] [1043.89][S02]to the sequence of actions that you took in proportion to how[1048.45] [1048.45][S02]unexpected it was, based on when you made that move,[1052.15] [1052.15][S02]did you think it was a good idea.[1053.53] [1053.53][S02]And then you won.[1055.24] [1055.24][S02]So maybe you should increase your idea of that[1057.57] [1057.57][S02]was a good idea a little bit.[1058.84] [1058.84][S02]Or maybe you lost, and you should decrease your sense[1061.695] [1061.695][S02]that that was a good idea a little bit.[1064.26] [1064.26][S02]And so that's kind of the main idea[1066.39] [1066.39][S02]behind reinforcement learning.[1067.66] [1067.66][S02]And it's a quite effective technique,[1069.34] [1069.34][S02]especially in environments where it's very unclear immediately[1073.75] [1073.75][S02]whether that was a good idea.[1075.23] [1075.23][S01] Yeah, absolutely.[1076.53] [1076.53][S02] In contrast, supervised learning[1078.37] [1078.37][S02]is where you have an input and you have a ground truth output.[1082.94] [1082.94][S02]So the classic example is you have a bunch of images,[1086.17] [1086.17][S02]and each image has been labeled with one[1088.12] [1088.12][S02]of a bunch of categories.[1089.485] [1089.485][S02]So there's an image--[1091.48] [1091.48][S02]car.[1092.27] [1092.27][S02]There's another image-- ostrich.[1094.22] [1094.22][S02]Another image-- a pomegranate.[1098.08] [1098.08][LAUGHTER][1100.24] [1100.24][S02]If you have a rich set of categories.[1102.88] [1102.88][S01] Exactly.[1103.93] [1103.93][S01]Tell me.[1104.51] [1104.51][S01]When you were here at DeepMind, and you[1106.135] [1106.135][S01]decided that you would do the acquisition, was Demis nervous?[1109.79] [1109.79][S02] I don't know if he was nervous.[1111.54] [1111.54][S02]I mean, I think that I said, well, I've[1115.92] [1115.92][S02]seen all these nice presentations,[1117.4] [1117.4][S02]but can I look it a little bit of the code?[1119.61] [1119.61][S02]And so-- because I wanted to make[1121.53] [1121.53][S02]sure there was real code behind it[1123.12] [1123.12][S02]and see what the coding standards were like,[1127.002] [1127.002][S02]did people actually write comments,[1128.46] [1128.46][S02]and that kind of thing.[1129.418] [1129.418][S02]So Demis kind of was a little unsure about this.[1133.39] [1133.39][S02]I said, oh, it doesn't have to be super secret code.[1136.03] [1136.03][S02]Just pick some little bit of code and show it to me.[1138.6] [1138.6][S02]And so I went into an office with one of the engineers,[1141.69] [1141.69][S02]and we kind of sat down for 10 minutes.[1143.67] [1143.67][S02]And I said, OK, what does this code do?[1146.83] [1146.83][S02]And, oh, OK, that thing, what does that do?[1149.35] [1149.35][S02]And can you show me the implementation of that?[1152.25] [1152.25][S02]And I came out satisfied.[1153.875] [1153.875][S01] It was neat and tidy.[1155.25] [1155.25][S02] It was reasonably neat and tidy.[1157.042] [1157.042][S02]I mean, for a small company trying to move quickly,[1159.97] [1159.97][S02]it was kind of researchy code.[1162.72] [1162.72][S02]But it was clearly interesting and well-documented.[1168.89] [1168.89][S02]And so--[1169.39] [1169.39][S01] I've heard that when you do your code,[1171.473] [1171.473][S01]you put a little thing, which is LGTM.[1174.213] [1174.213][S02] Oh yeah, it looks good to me.[1175.88] [1175.88][S02]Yeah, I use that in real life, too, not just for code reviews.[1179.51] [1179.51][S01] So in these presentations then,[1181.4] [1181.4][S01]can you remember what your impression was?[1183.993] [1183.993][S02] Yeah, I mean, it seemed[1185.41] [1185.41][S02]like they were doing really interesting work, particularly[1188.65] [1188.65][S02]in the reinforcement learning side.[1190.67] [1190.67][S02]We were focused on scaling, so we were training models[1193.84] [1193.84][S02]that were much, much bigger than the ones DeepMind was[1197.47] [1197.47][S02]playing with at the time.[1198.97] [1198.97][S02]But they were learning to use reinforcement learning[1202.24] [1202.24][S02]to sort of solve kind of game play, which[1206.157] [1206.157][S02]is a nice, clean environment for reinforcement learning.[1208.49] [1208.49][S02]But it seemed like the combination of reinforcement[1211.63] [1211.63][S02]learning plus a lot of the scaling work[1213.58] [1213.58][S02]that we had been working on would be a really good one.[1216.4] [1216.4][S01] Because it's like I guess[1217.24] [1217.24][S01]you're sort of approaching a problem from two[1219.115] [1219.115][S01]different directions-- like, really tiny with reinforcement[1221.8] [1221.8][S01]learning, really small, like toy models and building up.[1224.24] [1224.24][S01]And then you're kind of at this very, very big scale[1226.57] [1226.57][S01]with this sort of rich understanding,[1230.21] [1230.21][S01]understanding in inverted commas.[1231.82] [1231.82][S02] Yeah.[1232.51] [1232.51][S01] But then it's sort of putting the two together[1234.58] [1234.58][S01]where things become really powerful.[1236.03] [1236.03][S02] Yeah, yeah, indeed.[1237.28] [1237.28][S02]And that was a lot of the motivation[1240.73] [1240.73][S02]behind the combination we did last year[1244.0] [1244.0][S02]of the kind of legacy DeepMind and legacy Brain[1248.56] [1248.56][S02]and other parts of Google research.[1250.52] [1250.52][S02]We decided we would just combine the units together and form[1253.75] [1253.75][S02]Google DeepMind.[1255.13] [1255.13][S01] Yeah, and Gemini as the--[1256.99] [1256.99][S02] And Gemini, which actually predated[1259.03] [1259.03][S02]the idea of combining but really was like,[1261.5] [1261.5][S02]hey, we should really all work together on these problems[1264.25] [1264.25][S02]because we're all kind of sniffing around[1267.43] [1267.43][S02]the same kind of general direction of trying[1271.0] [1271.0][S02]to train really high-quality, large-scale, multimodal models.[1274.76] [1274.76][S02]And it doesn't make sense to fragment our ideas[1277.75] [1277.75][S02]and not work together and fragment our compute resources[1280.51] [1280.51][S02]and so on.[1281.03] [1281.03][S02]We should really just put all this together,[1283.72] [1283.72][S02]build a combined team to go after this problem,[1287.45] [1287.45][S02]and that's what we did.[1288.74] [1288.74][S01] So why Gemini?[1289.84] [1292.418][S02] I actually named it.[1293.71] [1293.71][S01] Did you?[1294.16] [1294.16][S02] Yeah.[1294.54] [1294.54][S01] I mean, you're very-- after the--[1295.95] [1295.95][S02] I do like naming things.[1296.85] [1296.85][S01] DistBelief.[1297.09] [1297.09][S02] It's fun.[1297.25] [1297.25][S02]Yeah, DistBelief.[1297.57] [1297.57][S01] It's your--[1298.89] [1298.89][S02] So Gemini relates to twins,[1301.92] [1301.92][S02]and I felt it was a good name for the twins of legacy DeepMind[1306.57] [1306.57][S02]and legacy Brain kind of coming together[1309.15] [1309.15][S02]to really start working together on an ambitious, multimodal[1313.465] [1313.465][S02]project.[1313.965] [1313.965][S01] I guess also Gemini--[1315.585] [1315.585][S01]I'm just thinking of the space missions.[1318.473] [1318.473][S02] Yeah.[1319.14] [1319.14][S01] It's like a precursor to Apollo.[1320.84] [1320.84][S02] Yeah, a good thing about a name that[1322.798] [1322.798][S02]has multiple meanings is--[1324.21] [1324.21][S02]so that was another reason to pick the name.[1326.47] [1326.47][S02]It's sort of the precursor to ambitious space program[1331.8] [1331.8][S02]progress.[1332.405] [1332.405][S01] So I want to come on to the multimodal stuff.[1334.78] [1334.78][S02] OK.[1335.363] [1335.363][S01] Just before I do, I guess[1337.26] [1337.26][S01]one of the big reasons why this big change has happened[1341.07] [1341.07][S01]in the sort of public consciousness of chat bots[1343.2] [1343.2][S01]and large language models is in part because of some work[1346.68] [1346.68][S01]that came out of Google Brain with Transformers.[1349.87] [1349.87][S01]If you'll sort of forgive the pun,[1351.757] [1351.757][S01]can you tell us a little bit about that Transformer work[1354.09] [1354.09][S01]and how transformative it's been?[1355.81] [1355.81][S02] Sure, yeah.[1356.98] [1356.98][S02]So it turns out a lot of the problems[1359.58] [1359.58][S02]you want to deal with in language[1361.95] [1361.95][S02]and in a bunch of other domains are problems of sequences.[1365.71] [1365.71][S02]So if you think about autocomplete in Gmail--[1369.987] [1369.987][S02]you're typing a sentence, and can the system[1371.82] [1371.82][S02]help you by finishing your sentence[1375.36] [1375.36][S02]or your thought for you?[1377.43] [1377.43][S02]A lot of that relies on seeing part of a sequence[1380.37] [1380.37][S02]and then predicting the rest of it.[1382.35] [1382.35][S02]And essentially, that's what these large language[1385.08] [1385.08][S02]models are trained to do.[1387.07] [1387.07][S02]They're trained to take in data one word[1389.97] [1389.97][S02]or one piece of a word at a time and then predict[1392.91] [1392.91][S02]what is the next thing that will follow.[1395.24] [1395.24][S01] Like fancy autocomplete.[1396.74] [1396.74][S02] Like fancy autocomplete.[1398.38] [1398.38][S02]Yeah, it turns out to be useful.[1399.732] [1399.732][S02]You can model a lot of different problems this way as well.[1402.19] [1402.19][S02]So translation you can model as taking[1405.75] [1405.75][S02]in the English version of a sentence[1407.85] [1407.85][S02]and then training the model to then output[1410.28] [1410.28][S02]the French version of the sentence when you have[1413.85] [1413.85][S02]enough English-French sentence pairs to sort of train[1417.36] [1417.36][S02]as like a sequence.[1419.28] [1419.28][S02]You can also use this in healthcare settings,[1421.655] [1421.655][S02]like if you're trying to predict a patient in front[1426.18] [1426.18][S02]of you is reporting these symptoms[1427.62] [1427.62][S02]and they have these lab test results, and in the past,[1429.87] [1429.87][S02]they've had these things, you can model that whole thing[1432.81] [1432.81][S02]as a sequence.[1433.77] [1433.77][S02]And then you can predict what are the likely diagnoses that[1437.94] [1437.94][S02]would make sense if you have other de-identified data[1442.17] [1442.17][S02]that you can train on that has also kind of been organized[1445.5] [1445.5][S02]in these sequences.[1446.74] [1446.74][S02]And the way you can do that is you[1448.77] [1448.77][S02]just hide the rest of the sequence,[1450.84] [1450.84][S02]and you force the model to try to predict what happens next.[1454.412] [1454.412][S02]It's quite an interesting thing that it's[1456.12] [1456.12][S02]so applicable to language translation, healthcare[1459.93] [1459.93][S02]settings, DNA sequences, all kinds of things.[1462.915] [1462.915][S01] But it's about the bit that you're paying attention[1465.54] [1465.54][S01]to at any point in time.[1467.2] [1467.2][S02] Yeah, so the models that were successful[1470.83] [1470.83][S02]prior to the Transformer architecture[1473.17] [1473.17][S02]were what are called recurrent models, where[1477.22] [1477.22][S02]they have some internal state.[1479.05] [1479.05][S02]And every time they see a word, they[1481.72] [1481.72][S02]do some processing to update their internal state.[1484.36] [1484.36][S02]And then they go on to the next word,[1486.25] [1486.25][S02]and then they do that again.[1487.85] [1487.85][S02]So now, they move their state forward a bit[1491.2] [1491.2][S02]and update the state with respect to the next word[1494.74] [1494.74][S02]that they just saw.[1495.68] [1495.68][S02]And so you can imagine this as like a 12-word sentence.[1499.49] [1499.49][S02]You're doing that updating of the state 12 times.[1502.37] [1502.37][S02]But every step is dependent on the previous one.[1505.25] [1505.25][S02]And so that means it's actually quite hard to get it to run fast[1508.36] [1508.36][S02]because you have this-- what's called a sequential dependency[1511.51] [1511.51][S02]where step seven depends on step six,[1514.13] [1514.13][S02]step six depends on step five, and so on.[1517.39] [1517.39][S02]So one of the things a collection of researchers[1520.45] [1520.45][S02]within Google Research did is they came up[1524.2] [1524.2][S02]with a pretty interesting idea, which is instead of just having[1527.5] [1527.5][S02]a single state that we update at every word,[1530.54] [1530.54][S02]let's process all the words all at once,[1533.95] [1533.95][S02]and let's remember the state that we get when[1536.86] [1536.86][S02]we're processing every word.[1538.84] [1538.84][S02]And then when we're trying to predict a new word,[1541.28] [1541.28][S02]let's pay attention to all of the previous states[1545.11] [1545.11][S02]and figure out how to learn to pay attention[1547.72] [1547.72][S02]to the important parts--[1549.05] [1549.05][S02]so that's the learned attention mechanism in Transformer--[1553.09] [1553.09][S02]in order to predict the next word.[1554.66] [1554.66][S02]And for some words, you might need to pay attention[1556.96] [1556.96][S02]to the previous word a lot.[1558.91] [1558.91][S02]For some contexts, it's very important[1561.82] [1561.82][S02]to pay attention kind of a little bit[1563.38] [1563.38][S02]to a lot of the words in the context.[1566.5] [1566.5][S02]But the important thing about that is it really[1568.51] [1568.51][S02]can be done in parallel.[1569.78] [1569.78][S02]You can take in 1,000 words, compute the state for each one[1573.91] [1573.91][S02]of them in parallel, and then that makes it sort of 10 to 100[1579.7] [1579.7][S02]times more efficient in terms of scaling and performance than[1585.22] [1585.22][S02]the previous recurrent models.[1586.53] [1586.53][S02]And so that was why that was such a big advance.[1588.53] [1588.53][S01] But then I guess there's other things that[1590.74] [1590.74][S01]seem to emerge from this.[1591.782] [1591.782][S01]I mean, sort of a conceptual understanding or maybe sort[1594.34] [1594.34][S01]of abstraction that's possible just through sequencing[1597.4] [1597.4][S01]and language alone--[1598.555] [1598.555][S01]I mean, was that a surprise?[1600.722] [1600.722][S02] Yeah, I mean, I think some of the earliest work we did[1603.43] [1603.43][S02]on language modeling in the Google Brain team was really[1606.4] [1606.4][S02]about modeling words not as their surface form of like[1610.75] [1610.75][S02]H-E-L-L-O or C-O-W but really about a high-dimensional vector[1617.89] [1617.89][S02]that represents kind of the way in which that word is used.[1621.95] [1621.95][S02]We're used to thinking in two and three dimensions as humans.[1625.13] [1625.13][S02]But when you have 100 dimensions or 1,000 dimensions,[1628.21] [1628.21][S02]there's a lot of room in 1,000-dimensional space.[1631.28] [1631.28][S02]But when you have things that are nearby,[1633.37] [1633.37][S02]and you've trained the model in such a way[1635.17] [1635.17][S02]that cow and sheep and goat and pig are all near each other,[1641.26] [1641.26][S02]and they're very far apart from espresso machine--[1643.905] [1643.905][LAUGHTER][1645.22] [1645.22][S01] Although milk could be in between.[1647.525] [1647.525][S02] Milk would probably be nearer the cow but kind[1649.9] [1649.9][S02]of in between the two.[1650.9] [1650.9][S01] Yeah.[1651.608] [1651.608][S02] It would probably be kind of on that 100-dimensional[1654.91] [1654.91][S02]line in 100-dimensional space.[1658.69] [1658.69][S02]So this is kind of why these models have surprisingly[1662.38] [1662.38][S02]powerful capabilities, I think, is because they're representing[1666.25] [1666.25][S02]things with so many high dimensions[1668.08] [1668.08][S02]that they can actually really latch on[1670.21] [1670.21][S02]to many different facets of a word[1673.3] [1673.3][S02]or a sentence or a paragraph simultaneously because there's[1677.38] [1677.38][S02]so much room in their representation.[1679.305] [1679.305][S01] It's sort of extracted the grounding[1681.64] [1681.64][S01]that we ourselves have given language, I guess.[1683.75] [1683.75][S02] Yeah, I mean, when we hear a word,[1686.12] [1686.12][S02]we don't just think of the surface form of the word.[1688.52] [1688.52][S02]We think cow-- oh, that triggers a bunch of other things,[1691.66] [1691.66][S02]like milk or espresso machine or milking and calf and bull.[1699.14] [1699.14][S02]And one of the things we found with those early word[1702.73] [1702.73][S02]representations was that directions had meaning.[1705.41] [1705.41][S02]So if you think about present tenses of verbs, like walk,[1711.95] [1711.95][S02]you would go in the same direction in this[1713.96] [1713.96][S02]100-dimensional space to get from walk to walked as you would[1719.3] [1719.3][S02]get from run to ran as you would go from read to read.[1728.033] [1728.033][S01] Wow.[1728.7] [1728.7][S02] Yeah.[1729.26] [1729.26][S01] So it actually understands-- understands,[1731.195] [1731.195][S01]I keep using that word, and I don't mean it.[1733.17] [1733.17][S01]But there is some representation of tenses[1736.4] [1736.4][S01]within the structure of these--[1737.92] [1737.92][S02] Yeah, and it's just emerged[1739.55] [1739.55][S02]from the training process.[1741.36] [1741.36][S02]It's not something we told it to do.[1742.86] [1742.86][S02]It's just the training algorithm we used[1745.16] [1745.16][S02]and the fact that language has lots[1747.35] [1747.35][S02]of ways in which particular forms are used[1752.6] [1752.6][S02]caused that to emerge.[1753.6] [1753.6][S02]And you could also, for example, change[1755.6] [1755.6][S02]from male or female versions of words and vice versa.[1759.75] [1759.75][S02]So cow to bull is the same direction[1763.85] [1763.85][S02]as queen to king or man to woman, woman to man, and so on.[1769.1] [1769.1][S01] Amazing.[1770.06] [1770.06][S01]But this is still-- this is just with language[1772.04] [1772.04][S01]that we're talking about here.[1773.07] [1773.07][S02] Yeah.[1773.21] [1773.21][S01] And so, OK, tell me tell[1774.72] [1774.72][S01]how does the multimodal aspect of this change?[1780.2] [1780.2][S01]How does it make it different?[1781.82] [1781.82][S02] Yeah, because you're still[1787.59] [1787.59][S02]representing the input data in these high-dimensional spaces.[1791.513] [1791.513][S02]And it's really a matter of how do[1792.93] [1792.93][S02]you get from the pixels of an image,[1795.4] [1795.4][S02]say, into something where, ideally, you'd[1800.04] [1800.04][S02]like the multimodal model to have the same kind of thing[1804.42] [1804.42][S02]that we have.[1805.14] [1805.14][S02]When we see a cow, that triggers kind[1808.66] [1808.66][S02]of similar activations in our brain to reading the word cow,[1812.83] [1812.83][S02]to hearing a cow moo, right?[1817.405] [1817.405][S02]And you kind of want to train models[1819.19] [1819.19][S02]so that they have that joint meaning and representation,[1823.18] [1823.18][S02]regardless of the way they arrived at that input data.[1826.93] [1826.93][S02]So if they see a video of a cow walking through a field,[1830.53] [1830.53][S02]that should trigger a whole bunch[1832.09] [1832.09][S02]of things that are related to that[1833.62] [1833.62][S02]in the model, based on the activations[1836.14] [1836.14][S02]that the model has built over--[1839.8] [1839.8][S02]typically, these are very deep, layered models.[1842.53] [1842.53][S02]And so the lowest layers typically[1844.33] [1844.33][S02]have very simple representations.[1846.26] [1846.26][S02]And then the deeper, the higher layers in the model,[1849.38] [1849.38][S02]build on those representations and build[1851.41] [1851.41][S02]more interesting and complex combinations[1855.76] [1855.76][S02]of features and representations of be it words or images or--[1860.377] [1860.377][S01] So when you're saying multimodal[1862.21] [1862.21][S01]from the ground up, which is kind of a big phrase[1864.252] [1864.252][S01]that you hear about Gemini, it's not[1866.44] [1866.44][S01]that you've got the word section over here,[1868.735] [1868.735][S01]the pixel section over here, and you're translating[1870.86] [1870.86][S01]between one and the other.[1871.895] [1871.895][S02] Right.[1872.24] [1872.24][S01] But in the model itself,[1873.847] [1873.847][S01]those representations are there.[1875.18] [1875.18][S02] Yeah, very early in the models.[1876.81] [1876.81][S01] Does that make it harder at the beginning[1879.018] [1879.018][S01]when you're setting it up?[1880.2] [1880.2][S01]Does it make it more difficult to do?[1884.59] [1884.59][S02] Yeah, I mean, I think figuring out[1886.63] [1886.63][S02]how to integrate different modalities into the model[1889.45] [1889.45][S02]and how should you train a multimodal model is[1892.93] [1892.93][S02]more complex than a simpler, pure language[1896.38] [1896.38][S02]or pure character-based model.[1899.14] [1899.14][S02]But you get a lot of benefits from it[1901.36] [1901.36][S02]in that you get sometimes cross-modal transfer, where[1905.74] [1905.74][S02]now seeing visual stuff about cows[1910.06] [1910.06][S02]actually helps inform the language.[1912.67] [1912.67][S02]Maybe it'd had seen a bunch of descriptions[1914.83] [1914.83][S02]of cows in meadows or something, but now[1917.8] [1917.8][S02]it suddenly has seen images of that and videos of that.[1922.39] [1922.39][S02]And it's actually able to bring those representations together[1926.17] [1926.17][S02]in a way that makes kind of similar things[1929.98] [1929.98][S02]trigger inside the model, regardless[1932.14] [1932.14][S02]of whether you saw the word cow or kind of the image of a cow.[1936.688] [1936.688][S01] Give me an example of the type of situation[1938.98] [1938.98][S01]you see this being useful in in the future.[1943.51] [1943.51][S02] Well, I think it's already useful, which is good.[1948.28] [1948.28][S02]I mean, as one example, you want to be[1950.2] [1950.2][S02]able to take in an image of a handwritten, whiteboard,[1957.83] [1957.83][S02]worked-out math problem and say, did the student[1961.42] [1961.42][S02]get this problem right?[1962.72] [1962.72][S02]Right?[1963.27] [1963.27][S02]And so now, you need to really bring[1965.7] [1965.7][S02]in the multimodal capabilities in one example.[1968.92] [1968.92][S02]You need to actually do handwriting recognition,[1972.19] [1972.19][S02]understand from that, OK, it's a physics problem[1975.54] [1975.54][S02]that someone's written on the board,[1977.04] [1977.04][S02]and it's got maybe a picture of a skier going down a slope.[1980.2] [1980.2][S02]In one of the early Gemini tech reports,[1982.57] [1982.57][S02]we had this good example of a student who had worked[1986.25] [1986.25][S02]out a problem on a whiteboard.[1988.14] [1988.14][S02]And you could actually ask Gemini,[1993.21] [1993.21][S02]did the student get this problem right?[1995.05] [1995.05][S02]If not, where did they go wrong?[1997.12] [1997.12][S02]And can you explain how to solve the problem correctly?[1999.76] [1999.76][S02]And it was actually able to tell that the student had incorrectly[2003.32] [2003.32][S02]applied the formula for a skier going down a frictionless slope.[2009.29] [2009.29][S02]And instead, they used the hypotenuse[2011.51] [2011.51][S02]instead of the height.[2012.69] [2012.69][S02]And it said, oh, no, actually, you should have used this.[2015.27] [2015.27][S02]And here's the problem worked out.[2017.16] [2017.16][S02]And it did all that and recognized all the handwriting[2020.75] [2020.75][S02]and the fact that this was a physics problem.[2022.83] [2022.83][S02]This kind of physics knowledge that the model already[2025.25] [2025.25][S02]had sort of was the right thing to apply.[2028.372] [2028.372][S01] I mean, that's a really neat way that you could[2030.83] [2030.83][S01]use the existing model of Gemini in the existing[2033.47] [2033.47][S01]model of education, I guess.[2034.86] [2034.86][S02] Totally.[2035.12] [2035.12][S02]Yeah.[2035.51] [2035.51][S01] But I suppose, actually, these[2037.26] [2037.26][S01]are not kind of isolated systems from one another.[2040.23] [2040.23][S01]So in some ways, do you think that these multimodal models[2043.04] [2043.04][S01]will change the way that we have to do education, full stop?[2047.2] [2047.2][S02] I mean, I think the potential[2049.06] [2049.06][S02]for using AI tools to help education is really amazing.[2056.27] [2056.27][S02]And we're sort of just at the beginning of this journey[2058.659] [2058.659][S02]as a society, I think.[2060.04] [2060.04][S02]We know, for example, that educational outcomes of students[2064.57] [2064.57][S02]who get one-on-one tutoring from another person[2067.75] [2067.75][S02]are two standard deviations better than students[2071.59] [2071.59][S02]who have a traditional classroom setting of a teacher and 30[2075.639] [2075.639][S02]or so students.[2077.77] [2077.77][S02]So how could we get everyone to the point[2081.9] [2081.9][S02]where they feel like they have the benefit[2084.87] [2084.87][S02]of an educational tutor that's one on one,[2087.268] [2087.268][S02]understands what they know, understands what they[2089.31] [2089.31][S02]don't know, can help them learn in the way that they learn best?[2093.84] [2093.84][S02]That is the potential of AI in education.[2096.94] [2096.94][S02]And I think, really, we're not that far away from something[2101.46] [2101.46][S02]where you could point a Gemini model or a future Gemini model[2106.23] [2106.23][S02]at something, some piece of material, and say,[2108.91] [2108.91][S02]can you help me learn this?[2110.33] [2110.33][S02]Take the chapter six in your biology textbook or something.[2114.31] [2114.31][S02]And it's got a bunch of images.[2116.32] [2116.32][S02]It's got a bunch of text.[2118.29] [2118.29][S02]Maybe it's got a lecture video that you watched as well.[2121.77] [2121.77][S02]And then you can actually say, I really[2124.32] [2124.32][S02]don't understand this thing.[2125.68] [2125.68][S02]Can you help me understand it?[2126.94] [2126.94][S02]It can ask you questions.[2128.35] [2128.35][S02]You can ask it questions.[2131.372] [2131.372][S02]You can answer the questions.[2132.58] [2132.58][S02]It can assess are you right or wrong[2134.9] [2134.9][S02]and really guides you in your learning journey[2139.16] [2139.16][S02]because it's individualized.[2140.87] [2140.87][S02]And we should be able to get that to many, many people[2145.73] [2145.73][S02]around the world in not just English, in languages[2150.92] [2150.92][S02]spoken by--[2153.77] [2153.77][S02]hundreds and hundreds of languages all around the world.[2156.853] [2156.853][S01] I mean, so I take what[2158.27] [2158.27][S01]you said about the lots of different languages[2160.187] [2160.187][S01]and trying to make these as broadly available as possible.[2163.1] [2163.1][S01]But is there a danger of creating[2165.14] [2165.14][S01]a bit of a two-tier system here where, on the one hand,[2167.79] [2167.79][S01]people who have access to these tools, as you described,[2171.66] [2171.66][S01]get far better outcomes, accelerate[2173.57] [2173.57][S01]their own learning and their own productivity, and then[2176.51] [2176.51][S01]anybody who is not fortunate enough to have access[2178.64] [2178.64][S01]to the tools really struggles?[2180.87] [2180.87][S01]I mean, is that something that concerns you?[2183.04] [2183.04][S02] Yeah, I mean, I think there is definitely[2185.207] [2185.207][S02]a risk of creating two-tier systems.[2186.9] [2186.9][S02]I think what we should strive to do[2189.26] [2189.26][S02]is make these technologies as broadly[2191.99] [2191.99][S02]accessible and universally accessible, if we can,[2195.72] [2195.72][S02]for everyone and really try to lean into the strengths of what[2199.56] [2199.56][S02]that will do for society and to make it affordable or free[2205.11] [2205.11][S02]for people to take advantage of the capabilities for education.[2208.78] [2208.78][S02]For health care, I think, is another area that is hugely--[2213.45] [2213.45][S02]huge potential for AI to really make a big difference[2217.47] [2217.47][S02]in healthcare accessibility.[2219.582] [2219.582][S01] Go back to Gemini, if we can.[2221.29] [2221.29][S02] Sure.[2221.957] [2221.957][S01] OK, so I guess if you started off[2225.0] [2225.0][S01]with Google Search, factuality must[2227.1] [2227.1][S01]have been absolutely at the cornerstone of everything[2230.01] [2230.01][S01]that you cared about.[2230.92] [2230.92][S02] Yeah.[2231.587] [2231.587][S01] But Gemini, I mean, you work with it all the time.[2234.588] [2234.588][S01]I imagine you've seen it say some quite outlandish things.[2237.005] [2237.005][S02] Yeah.[2237.672] [2237.672][S01] How are you sort of squaring[2240.0] [2240.0][S01]that circle in your head of releasing,[2242.76] [2242.76][S01]perhaps, some of the need for absolute factuality[2245.82] [2245.82][S01]at all times?[2246.93] [2246.93][S02] Yeah, it's actually a tricky balance as a company[2249.63] [2249.63][S02]because we are, from our origins, a search-based company.[2254.35] [2254.35][S02]And as you say, providing accurate,[2257.22] [2257.22][S02]factual information is kind of the pinnacle[2260.22] [2260.22][S02]of a search-engine experience.[2262.5] [2262.5][S02]And I think we actually had built interesting large language[2265.71] [2265.71][S02]models internally that people enjoyed conversing with.[2271.68] [2271.68][S02]Actually, some of them were available[2275.52] [2275.52][S02]internally during the pandemic.[2277.25] [2277.25][S02]So people were all at home.[2278.59] [2278.59][S02]And you could actually see internal usage spike[2281.01] [2281.01][S02]during lunchtime because people were having conversations[2283.92] [2283.92][S02]with their virtual chatbot.[2285.87] [2285.87][S02]Because who else are you going to talk to[2288.12] [2288.12][S02]when you're home alone or whatever?[2290.82] [2290.82][S02]But these models are trained to predict plausible next tokens,[2297.88] [2297.88][S02]essentially.[2298.57] [2298.57][S02]So a token, you can think of it as a word or a piece of a word.[2302.5] [2302.5][S02]And so when you predict plausible next tokens,[2305.56] [2305.56][S02]that's a different thing than that is absolute truth, right?[2309.0] [2309.0][S02]It's a probabilistically plausible sentence.[2313.11] [2313.11][S02]And that's different than a fact.[2314.91] [2314.91][S02]And I think one of the things we realized over time[2318.03] [2318.03][S02]was these models can actually be quite useful even if they're not[2322.23] [2322.23][S02]100% factual.[2323.47] [2323.47][S02]And so I think realizing there's all these other use cases,[2326.85] [2326.85][S02]or can you summarize this slide deck in five bullets--[2331.33] [2331.33][S02]and yes, you could argue about is that fifth bullet[2334.22] [2334.22][S02]exactly right.[2335.03] [2335.03][S02]But it's still pretty useful to get[2336.85] [2336.85][S02]4.5 bullets that are factually accurate about the slide deck.[2341.08] [2341.08][S02]And we're striving to make it five factually accurate bullets.[2344.66] [2344.66][S02]But even without that, I think the utility of these models[2347.98] [2347.98][S02]is actually quite high.[2348.95] [2348.95][S01] Was it an uncomfortable realization?[2350.95] [2350.95][S01]Because, of course, other labs did[2352.12] [2352.12][S01]push out their models earlier.[2353.543] [2353.543][S02] Yeah, yeah.[2354.46] [2354.46][S01] Do you think that guys had an abundance of caution[2358.36] [2358.36][S01]because of this factual issue?[2360.14] [2360.14][S02] I mean, I think we had[2361.96] [2361.96][S02]a number of different concerns, factuality being one of them.[2365.17] [2365.17][S02]Toxicity and bias in the way the model is trained[2369.64] [2369.64][S02]and the outputs that it can produce[2371.83] [2371.83][S02]is an area where we want to make the model less[2376.48] [2376.48][S02]biased in a lot of ways.[2378.43] [2378.43][S02]And so there were a whole number of areas[2381.28] [2381.28][S02]where we wanted to sort of be relatively[2383.92] [2383.92][S02]cautious before releasing things to the general public.[2389.16] [2389.16][S02]And I think we've gotten a lot of those issues kind of sorted[2394.17] [2394.17][S02]out enough that we think the products we've put out[2396.87] [2396.87][S02]in this space are useful even though there's obviously[2400.32] [2400.32][S02]room for improvement in things like factuality[2403.11] [2403.11][S02]and in bias and other areas.[2406.74] [2406.74][S02]So I think that's taken a little bit of an adjustment[2409.5] [2409.5][S02]for people-- is strive for the best you can be but also realize[2414.72] [2414.72][S02]that by not releasing something, you're sort of holding back[2419.34] [2419.34][S02]something that could be useful for a lot of people[2422.22] [2422.22][S02]even with its sort of foibles.[2424.253] [2424.253][S01] But then with those foibles,[2425.92] [2425.92][S01]then, so in which direction do we go from here?[2428.62] [2428.62][S01]Do you think that--[2429.42] [2429.42][S01]I mean, it sort of seems to me that there's[2431.212] [2431.212][S01]been this real shift in the way that computing happens,[2435.17] [2435.17][S01]as it were.[2435.67] [2435.67][S01]You get a calculator, you put in the same sum twice,[2438.01] [2438.01][S01]you get the same answer twice.[2439.05] [2439.05][S02] Yeah.[2439.41] [2439.41][S01] Whereas we're now in an era[2441.18] [2441.18][S01]of probabilistic computing.[2443.35] [2443.35][S01]And so I wonder whether--[2445.95] [2445.95][S01]is it that the public has to come to terms with that[2448.33] [2448.33][S01]and sort of accept that we're in an era where[2452.555] [2452.555][S01]things are much more human-like in that they can make mistakes?[2455.18] [2455.18][S01]Or is it something that you think is fixable?[2458.27] [2458.27][S02] I think it's some of both, right?[2460.52] [2460.52][S02]I mean, I think there's a bunch of sort[2463.455] [2463.455][S02]of technical approaches to some of these problems[2466.22] [2466.22][S02]that will make the factuality area issues better.[2470.09] [2470.09][S02]One instance is if you think about the data[2473.37] [2473.37][S02]the model is trained on--[2475.16] [2475.16][S02]trillions of tokens of text and other data[2478.97] [2478.97][S02]that are then mixed together in this giant soup of billions[2483.26] [2483.26][S02]and billions of parameters--[2486.23] [2486.23][S02]I like to think of that as like you've seen a lot of stuff,[2490.05] [2490.05][S02]but you don't recall it very well.[2493.25] [2493.25][S02]Whereas if you take information that is in--[2496.73] [2496.73][S02]one of the things we've been pushing on in Gemini[2499.22] [2499.22][S02]is having a long context window.[2501.72] [2501.72][S02]So when you have a long bit of space[2504.38] [2504.38][S02]where you can put a lot of direct information[2506.87] [2506.87][S02]that you're trying to summarize or manipulate or compare[2511.49] [2511.49][S02]in various ways or extract information[2513.38] [2513.38][S02]from, that information in the context window, the model[2516.83] [2516.83][S02]actually has a much clearer view of, right?[2519.15] [2519.15][S02]It's got the actual text and the representations[2522.15] [2522.15][S02]of that text not tangled together[2525.168] [2525.168][S02]with everything else it's seen.[2526.46] [2526.46][S01] So this context window is sort of the bit[2528.668] [2528.668][S01]that the model can see as important at that moment.[2532.507] [2532.507][S02] Yeah, it can sort of reason[2534.09] [2534.09][S02]about that in more fidelity than other things[2537.45] [2537.45][S02]that it's seen in its training process.[2539.59] [2539.59][S02]So it can take five PDFs of scientific articles,[2544.123] [2544.123][S02]and then you can ask questions about it,[2545.79] [2545.79][S02]like can you please tell me common themes[2548.97] [2548.97][S02]across these articles?[2550.0] [2550.0][S02]And then it's actually able to do that because it[2552.48] [2552.48][S02]has its own representation of all[2556.44] [2556.44][S02]the contents of those articles.[2557.95] [2557.95][S02]And that's one of the reasons we've[2559.408] [2559.408][S02]been pushing a lot on very long context windows[2562.02] [2562.02][S02]for Gemini models is we think that's a really[2564.18] [2564.18][S02]useful capability for factuality,[2566.68] [2566.68][S02]for video summarization, for all kinds of things.[2570.12] [2570.12][S01] Is there a limit, though, to the context window?[2572.62] [2572.62][S01]Can you just push and push and push and push[2574.453] [2574.453][S01]until it's sort of an infinite context window?[2577.47] [2577.47][S02] That is an excellent question.[2579.39] [2579.39][S02]Currently, the computational aspects of the attention process[2584.49] [2584.49][S02]are quite expensive.[2585.63] [2585.63][S02]So the longer you try to make it, the more expensive it gets.[2589.232] [2589.232][S01] Expensive in terms of time but also computing--[2591.69] [2591.69][S02] Compute, time--[2592.47] [2592.47][S01] --and eventually money.[2593.928] [2593.928][S02] Time and money and compute and all kinds of things.[2597.45] [2597.45][S02]But we think it may be possible to sort of come up with[2602.34] [2602.34][S02]algorithmic improvements that enable you to go beyond[2605.31] [2605.31][S02]the 2-million token context window,[2608.337] [2608.337][S02]which is what we have now.[2609.42] [2609.42][S02]I mean, a million tokens is quite a lot.[2611.37] [2611.37][S02]A million tokens is about 600 pages of text.[2615.54] [2615.54][S02]So that's like most books, 20 articles.[2620.67] [2620.67][S02]It's an hour of video.[2621.713] [2621.713][S01] How about on the other side?[2623.38] [2623.38][S01]Because you said it was a little bit of both,[2624.89] [2624.89][S01]that perhaps people have to adjust their expectations.[2627.14] [2627.14][S02] Yeah, so I think these models are tools,[2631.47] [2631.47][S02]and people need to understand the capabilities of their tools[2635.56] [2635.56][S02]but also some of the ways in which you probably[2638.61] [2638.61][S02]don't want to use the tool.[2641.99] [2641.99][S02]So, I mean, I think it's a bit of a educational process[2648.1] [2648.1][S02]for people.[2651.13] [2651.13][S02]Don't just trust every fact that comes out of a language model[2655.66] [2655.66][S02]right off the bat.[2656.81] [2656.81][S02]You need to apply a bit of scrutiny to that.[2661.51] [2661.51][S02]Sort of like--[2663.256] [2663.256][S02]I think we've taught people these days[2666.22] [2666.22][S02]that if you see something online,[2668.9] [2668.9][S02]that doesn't necessarily make it true.[2670.81] [2670.81][S02]I think a similar degree of skepticism[2672.985] [2672.985][S02]for some kinds of things from language models[2674.86] [2674.86][S02]is probably also appropriate.[2676.72] [2676.72][S02]That skepticism may decrease over time as the models improve,[2681.73] [2681.73][S02]but it's good to take it with a healthy dose of,[2685.6] [2685.6][S02]oh, that might not actually be true.[2689.453] [2689.453][S01] Aside from the context[2690.87] [2690.87][S01]windows, are there ways that you can[2694.5] [2694.5][S01]yourself, when you're sort of writing in prompts, sort of[2697.69] [2697.69][S01]minimize the risk of ending up with something that's[2701.37] [2701.37][S01]a complete hallucination?[2702.64] [2702.64][S02] So one technique that Google researchers kind of[2707.61] [2707.61][S02]came up with is what's called chain of thought prompting.[2711.22] [2711.22][S02]So in the same way if you just give the model[2716.19] [2716.19][S02]a sort of interesting math problem,[2718.17] [2718.17][S02]and you say, OK, what's the answer, it may get it correct,[2724.44] [2724.44][S02]but it may not.[2725.9] [2725.9][S02]And if instead, you say, here's an interesting math problem,[2730.41] [2730.41][S02]can you please show your work step by step--[2732.747] [2732.747][S02]so if you remember back to your fourth grade math teacher--[2735.205] [2735.205][S01] Oh, I do.[2735.77] [2735.77][S02] --he or she was probably saying,[2737.562] [2737.562][S02]you should really show your work step by step[2739.92] [2739.92][S02]and then get to the final answer and then[2741.933] [2741.933][S02]write the final answer down.[2743.1] [2743.1][S02]And that's partly because that helps[2746.6] [2746.6][S02]you get through that multi-step thinking process of how[2750.23] [2750.23][S02]do you actually go from what's being asked to, OK,[2753.48] [2753.48][S02]I need to calculate this and calculate this based on that[2756.48] [2756.48][S02]and so on and finally get to the answer.[2758.85] [2758.85][S02]And It turns out that not only makes the model's output more[2765.66] [2765.66][S02]interpretable because it kind of tells you[2767.73] [2767.73][S02]what steps it's going through, but it also[2770.632] [2770.632][S02]makes it more likely to get the correct answer.[2772.59] [2772.59][S01] What if it's not a math problem, though?[2774.757] [2774.757][S02] Yeah, I mean, even in non-crisply[2777.87] [2777.87][S02]defined right answer things, domains,[2782.02] [2782.02][S02]this approach kind of works.[2783.78] [2783.78][S02]And there's a bit of subtlety.[2786.66] [2786.66][S02]And I think people need to actually learn[2789.45] [2789.45][S02]how to use these models.[2790.93] [2790.93][S02]And the way in which you prompt them[2793.17] [2793.17][S02]is actually a big differentiator in how high quality[2797.88] [2797.88][S02]the output is.[2798.69] [2798.69][S02]Like, if you say, summarize this,[2801.81] [2801.81][S02]that might lead to one outcome.[2803.31] [2803.31][S02]If you say, please summarize this and give me[2806.64] [2806.64][S02]five bullet points that highlight[2810.12] [2810.12][S02]the major important pieces of the article[2812.55] [2812.55][S02]and identify two cons that the author wrote down--[2817.78] [2817.78][S02]if you say that, that's a much clearer set[2820.213] [2820.213][S02]of instructions to what the model should[2821.88] [2821.88][S02]do than just summarize this.[2823.078] [2823.078][S01] So when we put these things together, then,[2825.37] [2825.37][S01]so sort of breaking down step-by-step processes[2828.77] [2828.77][S01]but also understanding more context[2831.91] [2831.91][S01]and the multimodal stuff, too, are we[2834.31] [2834.31][S01]moving towards a situation where these kind of multimodal models[2837.82] [2837.82][S01]will understand us as individuals and our preferences?[2840.97] [2840.97][S02] Yeah, I mean, I think--[2842.68] [2842.68][S02]what you really want, I think, is a sort[2845.62] [2845.62][S02]of very personal version of Gemini for you[2848.23] [2848.23][S02]that understands what it is you're trying to do right now[2851.87] [2851.87][S02]but also understands the context in which you're[2854.41] [2854.41][S02]trying to do that.[2855.76] [2855.76][S02]I'm vegetarian.[2856.76] [2856.76][S02]So if I'm asking Gemini about restaurant recommendations[2859.9] [2859.9][S02]in London, and it knew that I was a vegetarian,[2862.58] [2862.58][S02]it would recommend different things than if I was not.[2866.29] [2866.29][S02]And I think a general model that is serving[2869.95] [2869.95][S02]the needs of every person the same[2872.86] [2872.86][S02]is not going to be as good as one[2874.57] [2874.57][S02]that actually understands a lot about you and your context.[2879.29] [2879.29][S02]There are some kinds of queries you[2880.93] [2880.93][S02]might like to ask a model that you can't quite[2884.59] [2884.59][S02]do today with Gemini but you could imagine wanting to do.[2887.88] [2887.88][S02]Can you take the pictures I took on the hike last week[2891.16] [2891.16][S02]and make an illustrated storybook[2894.52] [2894.52][S02]for my kid's bedtime tonight?[2897.91] [2897.91][S02]And it would know where those pictures from on your hike were[2901.405] [2901.405][S02]and how to make an illustrated storybook that[2903.28] [2903.28][S02]would appeal to your child.[2904.787] [2904.787][S02]It would maybe know how old your child[2906.37] [2906.37][S02]was to make it age appropriate.[2909.08] [2909.08][S02]So I think-- you can't do that now,[2911.18] [2911.18][S02]but that could be something useful.[2914.36] [2914.36][S02]People would want-- you'd want people to opt in to that.[2917.43] [2917.43][S02]I think the more information you want the model to know and have[2922.55] [2922.55][S02]in context, I think the more you want to sort of have people[2927.72] [2927.72][S02]understand what is happening.[2930.915] [2930.915][S02]One of the things I think we'll be able to do[2932.79] [2932.79][S02]is not train a version of the model on that data[2936.22] [2936.22][S02]but just have the right information available in context[2939.27] [2939.27][S02]in order to sort of call upon it when generating responses.[2942.437] [2942.437][S02]And I think that would be pretty nice.[2944.02] [2944.02][S01] So then you've got like this general structure[2948.57] [2948.57][S01]that you can almost imprint your own context onto.[2952.29] [2952.29][S01]But then that's kind of private for you.[2953.98] [2953.98][S02] That's right.[2954.64] [2954.64][S01] Nice.[2954.96] [2954.96][S02] Yeah.[2955.17] [2955.17][S02]That seems like it'd be pretty good.[2956.46] [2956.46][S01] Yeah, that would be pretty good.[2958.33] [2958.33][S01]Are we limited here to just audio and visual and things[2962.987] [2962.987][S01]that you can see on a screen-- language, whatever?[2965.07] [2965.07][S01]Or do we ever expect that these kind of assistants[2968.49] [2968.49][S01]will come out of our computers, as it were?[2971.398] [2971.398][S02] Yeah, I mean, I think there's actually[2973.44] [2973.44][S02]a lot of different kinds of new modalities of data that aren't[2978.09] [2978.09][S02]sort of strictly human modalities[2980.13] [2980.13][S02]that we want these models to understand,[2982.15] [2982.15][S02]so lots of temperature readings across the Earth[2986.74] [2986.74][S02]to help with weather prediction or genetic sequences[2990.37] [2990.37][S02]or LiDAR data for autonomous vehicles or robotics[2994.78] [2994.78][S02]applications.[2996.02] [2996.02][S02]And then in on setting, you want these models to perhaps[3002.34] [3002.34][S02]be able to help with real-world robotics applications,[3006.52] [3006.52][S02]be able to talk to a robotic device,[3009.67] [3009.67][S02]give it sort of instructions in plain language.[3013.51] [3013.51][S02]Can you please go into the kitchen[3015.52] [3015.52][S02]and wipe the counter down and recycle the soda[3020.35] [3020.35][S02]can I left on the counter and then[3022.0] [3022.0][S02]bring me a bag of pistachios or something, right?[3025.665] [3027.227][S02]Robots have traditionally not been able to understand language[3029.81] [3029.81][S02]like that, but I think we're on the cusp of enabling[3032.84] [3032.84][S02]that kind of capability and then being able to have robots do[3036.89] [3036.89][S02]50 or 100 useful tasks, even in messy environments[3040.91] [3040.91][S02]like this room, rather than the traditional setting in which[3045.17] [3045.17][S02]robots have already been deployed in the world, which[3047.63] [3047.63][S02]is very controlled environments like factory, assembly line[3051.86] [3051.86][S02]kind of things, where they go from there to there.[3053.97] [3053.97][S02]And it's a very predictable thing.[3055.41] [3055.41][S01] We've been talking here[3056.15] [3056.15][S01]as assistants, these things as being sort of augmenting[3059.0] [3059.0][S01]human abilities in that way.[3060.48] [3060.48][S01]And I can see it in medical settings, in education settings.[3063.9] [3063.9][S01]But is there more that the multimodal aspect of this[3068.33] [3068.33][S01]offers us in terms of, I don't know,[3070.52] [3070.52][S01]like how we understand the world?[3072.2] [3072.2][S02] Yeah, I mean, I think what these models can[3074.45] [3074.45][S02]do now is often do a few steps of reasoning[3078.62] [3078.62][S02]to get from what you asked it to do in order[3081.23] [3081.23][S02]to accomplish something.[3082.6] [3082.6][S02]And I think as these models improve in capability,[3085.98] [3085.98][S02]you'll be able to sort of get models[3088.41] [3088.41][S02]to work with you to do much more complex tasks.[3092.55] [3092.55][S02]And it's sort of a difference between[3097.97] [3097.97][S02]can you order a bunch of chairs at the chair[3101.0] [3101.0][S02]rental place versus plan me a conference, right?[3104.935] [3104.935][S01] Mmm.[3106.64] [3106.64][S02] The latter is much higher level, much more complex.[3112.272] [3112.272][S02]The right model would kind of ask you a bunch of follow up[3115.46] [3115.46][S02]questions because there's ambiguity in there.[3117.39] [3117.39][S02]How many people are coming?[3118.64] [3118.64][S02]What's it about?[3121.01] [3121.01][S02]Just like a human.[3122.06] [3122.06][S01] What country are you in?[3122.51] [3122.51][S02] Yeah, what country are you in?[3123.72] [3123.72][S02]Where do you want to have it?[3125.36] [3125.36][S02]When?[3128.06] [3128.06][S02]And then we'd set off and actually[3130.67] [3130.67][S02]be able to accomplish a lot of the kind of things[3133.25] [3133.25][S02]that you might want done in order[3135.92] [3135.92][S02]to do that high-level goal.[3137.31] [3137.31][S01] But then if you have this sort[3139.22] [3139.22][S01]of conceptual link, sort of these conceptual links--[3142.823] [3142.823][S01]I'm going back to cow here, right?[3144.24] [3144.24][S01]And it understands pictures.[3146.39] [3146.39][S01]And it understands, I don't know, I guess, gravity,[3149.52] [3149.52][S01]having seen videos on the internet.[3153.193] [3153.193][S02] It probably watched introductory lectures[3155.36] [3155.36][S02]on physics.[3155.86] [3155.86][S01] Right?[3156.61] [3156.61][S01]Oh, wow.[3157.53] [3157.53][S01]OK, so it understand it from that perspective.[3159.533] [3159.533][S02] Yeah.[3160.2] [3160.2][S01] Yeah.[3160.908] [3160.908][S02] Also seeing a bunch of things falling.[3163.65] [3163.65][S01] OK, so then could you go in one day[3166.15] [3166.15][S01]and say, draw me the blueprint for a really efficient[3169.21] [3169.21][S01]aeroplane?[3170.23] [3170.23][S02] Yeah, I mean, I think one[3171.73] [3171.73][S02]of the things these models need to be partnered with[3176.45] [3176.45][S02]is some exploratory process.[3181.11] [3181.11][S02]And that exploratory process can come in the form of maybe[3185.33] [3185.33][S02]it doesn't need to give you an answer in 200 milliseconds.[3189.27] [3189.27][S02]Maybe you'd be happy with your airplane tomorrow, right?[3193.835] [3193.835][S02]And so I think at that point, then you have a lot more[3197.18] [3197.18][S02]freedom in how would you design systems[3199.16] [3199.16][S02]to be able to efficiently do things like that where they can[3202.46] [3202.46][S02]go off and try a few experiments maybe in a simulator[3206.66] [3206.66][S02]that they have access to, or maybe they[3208.64] [3208.64][S02]create a simulator for basic fluid dynamics or something.[3213.0] [3213.0][S02]And they try a bunch of designs.[3217.14] [3217.14][S02]Maybe they have some ideas about what airplane shapes make sense,[3221.54] [3221.54][S02]having seen a bunch of existing airplanes.[3224.81] [3224.81][S02]And so then they can kind of try to accomplish[3227.3] [3227.3][S02]what it is you asked.[3228.48] [3228.48][S02]Hopefully, they first asked you, well,[3230.61] [3230.61][S02]what characteristics do you want your airplane to have?[3233.105] [3233.105][S01] It was a paper airplane all along[3234.98] [3234.98][S01]is what I wanted.[3235.67] [3235.67][S02] Paper airplane.[3236.34] [3236.34][S02]Yeah, it's important to know if it's paper.[3238.41] [3238.41][S02]That reduces the cost a lot.[3242.07] [3242.07][S02]So I think those kinds of things will come eventually.[3247.3] [3247.3][S02]It's a little hard to tell exactly when[3249.57] [3249.57][S02]those capabilities-- that's a pretty complicated sort[3252.01] [3252.01][S02]of integration of what you want the reasoning in the model[3256.62] [3256.62][S02]to do, the knowledge it needs, what you're asking it to do,[3260.4] [3260.4][S02]and how you're asking it.[3261.85] [3261.85][S02]But we're already seeing pretty big advances in capabilities[3266.04] [3266.04][S02]of these models over 5-year, 10-year periods.[3269.68] [3269.68][S02]And so over a 5-year, 10-year period, that might be possible.[3274.44] [3274.44][S02]It might even be sooner than that for can you help me[3278.07] [3278.07][S02]design an airplane with these characteristics.[3280.23] [3280.23][S01] But I guess these are[3281.605] [3281.605][S01]like the early, early precursors to what we[3284.04] [3284.04][S01]might hope Apollo would be.[3286.02] [3286.02][S02] Yeah, exactly.[3287.14] [3287.14][S02]That's why it's Gemini.[3288.12] [3288.12][S01] That's why it's Gemini.[3289.68] [3289.68][S01]Amazing.[3290.37] [3290.37][S01]Jeff, thank you so much for joining me.[3292.18] [3292.18][S02] It's a pleasure to be here.[3293.08] [3293.08][S02]Thank you for having me.[3294.08] [3294.08][S01] In a lot of ways, I think Jeff's whole story[3296.49] [3296.49][S01]is one about scale.[3298.05] [3298.05][S01]For Google Search, it was about how do you[3300.09] [3300.09][S01]get more of the web, more users, faster queries.[3303.84] [3303.84][S01]For neural networks, it was about more computing power,[3306.97] [3306.97][S01]more machines.[3308.17] [3308.17][S01]And in the recent era of machine learning,[3310.39] [3310.39][S01]it's been about more and more and more data.[3314.19] [3314.19][S01]But something emerges from all of that--[3316.27] [3316.27][S01]a genuine conceptual model of the world, one[3319.38] [3319.38][S01]that is capable of abstraction and has already a proven ability[3323.7] [3323.7][S01]to enhance human productivity.[3326.44] [3326.44][S01]And it's telling that Jeff isn't finished there.[3329.08] [3329.08][S01]There is more to come, more sensors, more modes and, when[3332.79] [3332.79][S01]combined with the reinforcement learning tools that were born[3335.37] [3335.37][S01]in this building, maybe also more progress on the path[3339.36] [3339.36][S01]to AGI.[3341.49] [3341.49][S01]If you've enjoyed this episode, please[3343.47] [3343.47][S01]make sure that you subscribe to our podcast.[3345.39] [3345.39][S01]And if you have any feedback, or you[3347.04] [3347.04][S01]want to suggest a guest that you'd like to hear from, then[3349.8] [3349.8][S01]why not leave us a comment on YouTube?[3351.65] [3351.65][S01]Until next time.[3353.72]