[0.5][MUSIC PLAYING][4.0] [6.331][S03] Welcome to "Google DeepMind--[8.039] [8.039][S03]The Podcast" with me, your host, Professor Hannah Fry.[10.71] [10.71][S03]Now, 2025 is coming and so is the era of agentic AI,[15.09] [15.09][S03]although, of course, you'll have known about that months[18.35] [18.35][S03]ago if you've been listening to us.[20.4] [20.4][S03]So now listen in carefully as we tell you about the next thing.[25.53] [25.53][S03]It's called Project Astra, and it's a research prototype[28.61] [28.61][S03]that is pushing the boundaries of what[30.8] [30.8][S03]might be possible with a universal AI assistant.[35.16] [35.16][S03]It's an agent that is, by design, not necessarily tethered[39.41] [39.41][S03]to a particular device or a screen or a keyboard.[43.02] [43.02][S03]This is right at the brink of the cutting edge.[46.95] [46.95][S03]And today, we get to play with it.[49.74] [49.74][S03]Our Project Astra brings together all of the things[52.25] [52.25][S03]that we've spoken about in this series-- memory, vision,[55.8] [55.8][S03]context, reasoning, real-time interaction.[59.31] [59.31][S03]And someone who can tell us a lot about all of those,[62.78] [62.78][S03]and how he and his team have managed[64.81] [64.81][S03]to get them to work together, is Greg Wayne, Director[68.44] [68.44][S03]of Research at Google DeepMind.[70.25] [70.25][S03]Now, Greg also happens to be one of the very first people I[73.09] [73.09][S03]interviewed right at the very, very beginning of series[76.15] [76.15][S03]one of this podcast.[77.42] [77.42][S03]Greg, welcome back.[78.527] [78.527][S02] Hello, Hannah.[79.61] [79.61][S03] Let's start at the beginning, then.[81.568] [81.568][S03]What is Project Astra?[82.93] [82.93][S02] Project Astra is a team and a project[86.05] [86.05][S02]and a prototype aiming to build an AI assistant with eyes, ears,[92.15] [92.15][S02]and a voice that's co-present with you.[94.04] [94.04][S02]So it is with you in any place you are.[97.61] [97.61][S02]So either through smart glasses or your phone or your computer,[102.23] [102.23][S02]it can see what you're doing.[103.82] [103.82][S02]It can talk to you about it.[105.22] [105.22][S03] Like a little assistant[105.94] [105.94][S03]that sits on your shoulder.[107.21] [107.21][S02] Yeah, yeah, like a little parrot[109.12] [109.12][S02]on your shoulder that's hanging out with you[111.082] [111.082][S02]and talking to you about the world.[112.54] [112.54][S03] That's just smarter than you.[114.49] [114.49][S02] In some ways, yeah.[116.167] [116.167][S03] I suppose, then, in a way,[117.75] [117.75][S03]that that is different to Google Assistant or Gemini.[121.71] [121.71][S02] Yeah.[122.94] [122.94][S03] It's more embedded in the world.[125.04] [125.04][S02] Yeah.[125.748] [125.748][S02]So the older version of a Google Assistant was kind of like[130.53] [130.53][S02]a command-and-control system for your home or your information--[137.505] [137.505][S02]turn on this song on YouTube, or something like that.[140.88] [140.88][S02]And this is about being able to talk about the world.[145.47] [145.47][S02]It builds on Gemini.[146.83] [146.83][S02]Gemini is the intelligence underneath the hood,[149.7] [149.7][S02]along with some other systems.[151.74] [151.74][S02]I think it's complementary to the vision of Gemini, in a way,[156.37] [156.37][S02]and works with it and maybe helps shape Gemini and is also[159.33] [159.33][S02]shaped by it.[160.15] [160.15][S03] Can we try it out then?[160.84] [160.84][S03]Because I think this is something[161.64] [161.64][S03]we have to experience to understand.[163.41] [163.41][S02] Totally, yeah.[164.71] [164.71][S02]OK.[165.21] [165.21][S02]I'm going to turn it back on.[168.6] [168.6][S01] Hey, Greg.[169.95] [169.95][S03] It knows you.[171.07] [171.07][S02] It does.[171.903] [171.903][S02]Yeah, I've had many conversations with you before.[175.758] [175.758][S04] Indeed, we have spoken previously.[177.8] [177.8][S04]It is good to talk to you again.[179.33] [179.33][S02] I'm going to hand you to my friend[181.247] [181.247][S02]Hannah for one second.[182.565] [182.565][S01] All right.[183.44] [183.44][S01]It's nice to meet Hannah.[184.55] [184.55][S01]I'm here whenever you need me.[186.35] [186.35][S03] Hi, Project Astra.[187.85] [187.85][S03]OK.[188.41] [188.41][S04] Hello, Hannah.[189.618] [189.618][S03] Can you guess what we're doing today?[193.41] [193.41][S04] Based on the equipment and setup,[195.41] [195.41][S04]it appears you are recording a video or podcast in a studio.[198.59] [198.59][S04]Is this correct?[199.58] [199.58][S03] It absolutely is.[201.002] [201.002][S03]OK, I'm going to test you on a few things, all right?[203.21] [203.21][S03]Because I brought in a few objects.[205.15] [205.15][S03]Is that OK?[205.92] [209.71][S04] Yes.[212.26] [212.26][S04]I am ready when you are.[213.71] [213.71][S04]I look forward to seeing the objects.[215.39] [215.39][S03] OK.[216.21] [216.21][S03]All right.[216.71] [216.71][S03]Let's start off.[218.15] [218.15][S03]This is my favorite book, Project Astra.[219.91] [219.91][S03]Can you see the book?[220.97] [220.97][S04] Ah, yes.[223.76] [223.76][S04]I can see the book.[224.78] [224.78][S04]It displays a graphic with a repeating pattern, a motif known[227.98] [227.98][S04]as MC Escher's Circle Limit IV.[231.02] [231.02][S03] [GASPS] Can you guess what the book is?[235.067] [235.067][S04] Based on the image of the open page[237.15] [237.15][S04]and the artwork on it, it appears to be "Fermat's Last[239.78] [239.78][S04]Theorem"--[240.28] [240.28][S03] No.[241.5] [241.5][S02] [LAUGHS][243.33] [243.33][S03] OK.[244.09] [244.09][S03]I'm so-- I'm so impressed.[246.37] [246.37][S03]I'm so impressed.[248.02] [248.02][S03]OK, I brought in--[249.21] [249.21][S02] That's a crufty old copy, too.[251.8] [251.8][S03] I know.[253.165] [253.165][S03]You can tell it's my favorite book.[254.77] [254.77][S03]It's been loved.[255.64] [255.64][S03]It's been very loved over many years.[258.329] [258.329][S03]OK.[259.44] [259.44][S03]I'm going to try a couple of other things[261.734] [261.734][S03]just to see if I can really test you.[264.697] [264.697][S03]All right, let's try this.[265.78] [267.942][S04] It's good to see you[269.4] [269.4][S04]have brought out the nice furniture to complement[271.53] [271.53][S04]the book.[272.05] [272.05][S04]Do you require my assistance with anything else?[274.145] [274.145][S03] [LAUGHS] I have brought out the nice furniture,[276.45] [276.45][S03]Project Astra.[277.08] [277.08][S03]Yeah, thank you.[277.81] [277.81][S03]OK, here's-- what about this?[280.3] [280.3][S03]Do you know what that is?[282.06] [282.06][S04] It appears to be a model of a brain.[285.42] [285.42][S03] Which hemisphere?[288.66] [288.66][S04] It is the left hemisphere of the brain model.[291.16] [291.16][S02] Wow.[291.84] [291.84][LAUGHS][294.11] [294.11][S03] Why-- why did you say--[295.637] [295.637][S02] Well, I mean, I had to do a lot of mental rotation[298.22] [298.22][S02]for that.[299.527] [299.527][S03] Me, too.[300.36] [300.36][S03]Which way around is it?[301.41] [301.41][S03]Oh, yeah.[302.07] [302.07][S03]But got it right.[303.24] [303.24][S03]Amazing.[304.34] [304.34][S03]I might just try it because there's a whiteboard behind you[306.95] [306.95][S03]that has been there all--[309.03] [309.03][S03]I've basically been looking at this for many months now.[311.91] [311.91][S03]So Astra, if I show you that there,[314.43] [314.43][S03]tell me what some of the drawings are on the whiteboard.[317.1] [319.868][S04] The whiteboard contains a variety of drawings,[322.41] [322.41][S04]including a tree, buildings, and a series of connected lines[326.15] [326.15][S04]and shapes.[326.94] [326.94][S04]There are also some musical notes and text bubbles.[330.308] [330.308][S03] I'm very impressed.[331.6] [331.6][S03]I'm very impressed.[332.7] [332.7][S03]Let me just put this little guy back where it belongs over here.[337.28] [337.28][S03]So Project Astra is still this research prototype.[340.35] [340.35][S03]It's not available as a product that everyone can just download.[343.79] [343.79][S03]Why demo it now?[346.11] [346.11][S03]Why wouldn't you wait until it was ready?[348.117] [348.117][S02] I think it's nice to bring[349.7] [349.7][S02]the public along on the journey in a way, right?[351.85] [351.85][S02]I think people should get to know[354.36] [354.36][S02]what's sort of being developed inside the labs.[356.77] [356.77][S02]And we're giving it to more people to start playing with[360.33] [360.33][S02]and start adjusting to or giving feedback about.[363.04] [363.04][S02]Now it's a co-creation process where it's not[365.97] [365.97][S02]only some kind of thing that's being cooked up in a lab,[369.31] [369.31][S02]it's also being cooked up in collaboration[371.73] [371.73][S02]with a group of users around the world, people outside of Google.[377.56] [377.56][S02]That's important too.[378.49] [378.49][S02]So if we're going to make this really a helpful thing[381.27] [381.27][S02]for humanity, then people need to start using it and telling us[384.15] [384.15][S02]how they feel about it.[385.285] [385.285][S03] So have people been taking this out and about[387.66] [387.66][S03]and trying it out in the real world?[389.64] [389.64][S02] Yeah, we've had these trusted testers, people[394.8] [394.8][S02]who are using it, who were just signed up[396.84] [396.84][S02]to be kind of early adopters.[398.17] [398.17][S03] What are people using it for?[399.878] [399.878][S02] People are using it for things like getting[402.3] [402.3][S02]fashion advice from Astra--[404.372] [404.372][S03] Oh, really?[405.33] [405.33][S03]In what way?[406.39] [406.39][S02] Like, what would match with this?[409.73] [409.73][S02]So yeah, Astra is kind of just like a partner.[413.21] [413.21][S02]Oh, what do you think about--[414.68] [414.68][S02]how could I have a fresher look here?[417.65] [417.65][S03] Oh, wow.[419.09] [419.09][S03]I mean, that's a very clever parrot.[422.107] [422.107][S02] It's a very clever parrot.[423.69] [423.69][S03] But then what about hardware?[424.92] [424.92][S03]I mean, at the moment, as you say, it's on your smartphone.[427.77] [427.77][S03]But are we talking about, eventually, in glasses?[430.74] [430.74][S02] Yeah, but not only.[433.25] [433.25][S02]So I think when an earlier version of this project started,[436.35] [436.35][S02]it was really trying to tease out[440.06] [440.06][S02]how useful smart glasses would be if an AI was on them.[444.29] [444.29][S02]So in smart glasses, it's the most intimate and, in some ways,[448.94] [448.94][S02]amazing experience because you've got this--[451.4] [451.4][S02]you feel augmented personally, like you're[454.22] [454.22][S02]having a conversation with a smart version of yourself[456.742] [456.742][S02]that's just sitting there and telling you[458.45] [458.45][S02]whatever you want to know.[459.54] [459.54][S02]But the software stack effectively[461.21] [461.21][S02]is agnostic to how your--[464.43] [464.43][S02]I mean, there's specializations for each device,[467.31] [467.31][S02]but you can have it on phones or computers or VR headsets.[470.898] [470.898][S03] I was thinking as well, actually, as we were just[473.44] [473.44][S03]playing around with it, there's a potential benefit[475.565] [475.565][S03]for people who are partially sighted here or blind too,[478.26] [478.26][S03]right?[478.76] [478.76][S02] Yeah, that's an obsession of mine.[480.677] [480.677][S02]We've talked about this sort of AI[482.56] [482.56][S02]as being co-present or sharing your perspective.[485.33] [485.33][S02]And sometimes, you want another seeing and hearing[492.84] [492.84][S02]kind of intelligence with you, but you don't always need one.[497.83] [497.83][S02]So when are the cases when you want[499.99] [499.99][S02]a system that can see alongside you if you see but don't[505.57] [505.57][S02]understand or if you can't see?[510.13] [510.13][S02]And so that's a whole category.[512.33] [512.33][S02]And there's a lot of people out there, hundreds of millions[516.64] [516.64][S02]of people, who have vision impairment.[519.289] [519.289][S02]And what's the gold standard of help for that population?[524.27] [524.27][S02]Well, it's having someone by their side[526.73] [526.73][S02]who can help them out in the world.[529.58] [529.58][S02]And this technology sort of is able to replicate that[534.62] [534.62][S02]to a large extent.[535.53] [535.53][S02]We have more nascent ideas, too, about other kinds[540.5] [540.5][S02]of disabilities.[541.35] [541.35][S02]So you could imagine helping people[543.8] [543.8][S02]who have difficulty scanning emotions and faces,[547.17] [547.17][S02]understanding that in certain circumstances.[550.122] [550.122][S03] So people, potentially, with autism[552.08] [552.08][S03]could use this to help.[553.47] [553.47][S02] Yeah.[554.39] [554.39][S02]I wouldn't recommend it as a prescribed drug[556.79] [556.79][S02]right at the moment, but I think with further development,[560.43] [560.43][S02]it could definitely be.[561.77] [561.77][S02]Also for training yourself, since you[565.55] [565.55][S02]could work on understanding faces[567.62] [567.62][S02]and have Astra give you feedback,[570.333] [570.333][S02]like, hey, tell me about this.[571.92] [571.92][S02]I remember-- it's on a separate topic,[574.68] [574.68][S02]but when I was doing a homestay, I[577.67] [577.67][S02]was learning French one summer, and I couldn't[581.0] [581.0][S02]pronounce certain words.[582.35] [582.35][S02]Like, the difference between the word for "street"[584.44] [584.44][S02]and the word for "wheel," like "la rue" and "la roue."[589.705] [589.705][S02]I still can't do it, right?[591.11] [591.11][S02]But I sat there with my homestay brother,[594.61] [594.61][S02]and I was just trying to copy him for a while,[597.2] [597.2][S02]and he's just blew me off after a few minutes.[599.27] [599.27][S02]He's like, I'm not sitting there with you.[601.34] [601.34][S02]Astra would be infinitely patient with you[603.88] [603.88][S02]and could help you with that kind of thing.[606.23] [606.23][S02]Obviously, memory-- so we have a system[610.12] [610.12][S02]that has perfect in-session memory, we call it.[616.01] [616.01][S02]So when the camera is rolling, basically, it[618.25] [618.25][S02]remembers the last 10 minutes photographically,[622.51] [622.51][S02]but it will also remember what you've talked about in the past.[626.24] [626.24][S02]That's why it remembers I'm Greg.[628.66] [628.66][S02]And probably if we turn it back on we say,[631.155] [631.155][S02]do you remember who was talking to you besides Greg[634.34] [634.34][S02]the last time, it will remember Hannah.[638.08] [638.08][S02]So this could be used for people with some cognitive impairments,[642.25] [642.25][S02]too, at some point.[646.12] [646.12][S02]I think one of the things that we're[648.19] [648.19][S02]excited about, too, is this idea of proactiveness,[650.96] [650.96][S02]so it deciding on its own that you have a need[656.27] [656.27][S02]and then channeling the response to that need[661.75] [661.75][S02]without your actual need to give it a steer.[666.2] [666.2][S02]So for example, it could be a useful system[669.13] [669.13][S02]for reminding you of things, going through the memories[671.89] [671.89][S02]and saying, oh, don't forget, you need to pick this up[674.41] [674.41][S02]on your way home, or whatever.[675.86] [675.86][S03] So you're not necessarily just[677.99] [677.99][S03]proactively switching it on when you want to talk to it,[680.99] [680.99][S03]but it could be there in the background[683.05] [683.05][S03]and then it bring up something when[685.96] [685.96][S03]it thought it was appropriate.[687.442] [687.442][S02] Yeah, yeah.[688.4] [688.4][S02]So the idea is like, you are going home[692.83] [692.83][S02]and it's like, hey, don't forget that you need to pick up[696.837] [696.837][S02]some orange juice because you ran out this morning[698.92] [698.92][S02]or whatever.[699.42] [699.42][S03] Oh, wow, because it remembers having[701.42] [701.42][S03]seen that in the morning.[702.63] [702.63][S02] Yeah, exactly.[704.12] [704.12][S03] So, I mean, I guess at this stage,[705.45] [705.45][S03]this is like painting ideas of what's possible, right?[707.69] [707.69][S02] Yeah, we don't have that yet, no.[709.29] [709.29][S02]But that's the kind of thing that we could build next, yeah.[711.96] [711.96][S03] But you can see the beginnings of it in this.[716.522] [716.522][S02] Yeah.[717.23] [717.23][S02]I mean, so I could easily say, here's my fridge[721.13] [721.13][S02]and, oh, no, there's not much orange juice.[723.498] [723.498][S02]And then I'd say, hey, what do you think I should[725.54] [725.54][S02]get at the supermarket later?[726.9] [726.9][S02]And it would remember that, yeah.[729.11] [729.11][S02]But I would have to give it a bit more of a context or--[732.457] [732.457][S03] Hold its hand a little bit more, as it were.[734.79] [734.79][S02] Yeah.[735.498] [735.498][S03] Yeah.[736.37] [736.37][S03]Do you find yourself having to correct it a lot?[738.78] [738.78][S03]I mean, do you notice glitches?[742.43] [742.43][S02] Yeah, yeah.[743.49] [743.49][S02]So one thing it does once in a while is it[747.2] [747.2][S02]says it can't really see something[749.69] [749.69][S02]that's it can clearly see.[750.99] [750.99][S02]And so you'll be reading--[753.26] [753.26][S02]like on a bookshelf.[754.52] [754.52][S02]And you'll say, can you read the titles of the bookshelf?[757.88] [757.88][S02]And it'll say like, oh, no, the titles, I can't make--[760.69] [760.69][S02]and you'll say something like-- you'll kind of do a Jedi mind[763.29] [763.29][S02]trick on it.[763.84] [763.84][S02]You'll be like, yes, you can see.[766.13] [766.13][S02]And it'll be like, yes, I can.[767.38] [767.38][S02]And it'll then-- [LAUGHS][769.41] [769.41][S02]This is sort of a weird limitation of the--[773.825] [773.825][S02]yeah, the agreeableness is something[776.31] [776.31][S02]that you can influence.[777.43] [777.43][S03] Is it susceptible to encouragement, then?[779.903] [779.903][S02] Yes.[780.57] [780.57][S03] Really?[781.362] [781.362][S02] [LAUGHS][782.46] [782.46][S03] I mean, hey, it works for humans, too.[784.95] [784.95][S03]A little bit of encouragement, and suddenly you[787.14] [787.14][S03]can do things you didn't think were possible.[789.81] [789.81][S03]So what other kind of environments[792.3] [792.3][S03]does it struggle with?[793.665] [793.665][S03]I mean, it's quite quiet in here.[795.04] [795.04][S03]It's quite well lit.[797.74] [797.74][S03]There's not lots of busyness going on.[801.845] [801.845][S03]Does it work just as well in those kind of environments--[804.22] [804.22][S03]busy, noisy, dark, perhaps?[806.98] [806.98][S02] In some ways, operating in more environments[809.76] [809.76][S02]is an important thing that we need to develop,[812.64] [812.64][S02]in particular noise conditions.[814.93] [814.93][S02]So as I said to you, Astra really does hear.[820.43] [820.43][S02]It actually takes in the audio directly,[824.79] [824.79][S02]converts it into a system that is the neural networks of taken[829.91] [829.91][S02]sound and code them as some kind of a package of information[833.81] [833.81][S02]that is processed by the language[836.24] [836.24][S02]model, Gemini, directly.[839.67] [839.67][S02]But the system isn't really trained to identify[845.9] [845.9][S02]different voices, so it will have[849.65] [849.65][S02]trouble understanding your voice versus my voice[852.138] [852.138][S02]when we're talking.[852.93] [852.93][S02]So if there's other bystanders who are having conversations,[857.04] [857.04][S02]Astra will pick that up as potentially the user's speech.[864.53] [864.53][S02]Or it actually has a system that kind of wakes up[871.22] [871.22][S02]and listens for a bit when there is somebody[873.74] [873.74][S02]speaking with enough intensity.[877.09] [877.09][S02]And it will just start listening to errant speech and be confused[883.6] [883.6][S02]if there's nothing directed at it.[885.02] [885.02][S02]So yeah, noisy environments will confuse it.[887.32] [887.32][S03] When you say distinguish[888.82] [888.82][S03]between the different voices, as in the waveform itself?[895.66] [895.66][S02] So there's an old problem called the cocktail[898.84] [898.84][S02]problem, which is the whole problem of what's[904.15] [904.15][S02]more technically known as source separation.[906.47] [906.47][S02]So it's understanding one sound source from another.[911.6] [911.6][S02]So if there's a guitar and someone singing,[914.51] [914.51][S02]you could isolate that into two tracks, the guitar track[917.92] [917.92][S02]and the singing track.[920.23] [920.23][S02]Likewise, you might want to be able to distinguish[922.9] [922.9][S02]the one speaker's track and another speaker's track.[928.06] [928.06][S02]So that might be possible to do within the single modality[934.77] [934.77][S02]or sense of audio.[938.86] [938.86][S02]It would also be possible to do that in a multimodal sense,[943.24] [943.24][S02]integrating across senses.[944.98] [944.98][S02]So for example, when I know that it's you speaking,[948.58] [948.58][S02]I can also see the movement of your lips[950.34] [950.34][S02]rather than the movement of someone else's lips.[953.28] [953.28][S02]So ultimately, you could imagine the systems would[956.1] [956.1][S02]use all sorts of cues, even to change[959.58] [959.58][S02]the way they perceive a sound.[961.33] [961.33][S03] Because I guess this is, in some ways,[963.45] [963.45][S03]the thing that makes Project Astra so difficult,[966.07] [966.07][S03]but also the thing that gives it the potential,[969.16] [969.16][S03]because the cocktail problem, as you say,[971.5] [971.5][S03]which humans are extremely good at-- you're in a cocktail party[975.06] [975.06][S03]and you can hear exactly what the person next to you is saying[977.77] [977.77][S03]despite loads of voices going on all around--[979.9] [979.9][S02] I have trouble, actually.[980.85] [980.85][S03] Ah.[981.33] [981.33][S03]Actually, do you know what?[982.455] [982.455][S03]Honestly, so do I.[983.79] [983.79][S02] That's why it's a problem.[985.373] [985.373][LAUGHTER][986.79] [986.79][S03] But broadly speaking, humans[988.53] [988.53][S03]are quite good at these things.[990.16] [990.16][S03]And when you have the audio-only problem,[992.48] [992.48][S03]it's really hard to solve.[994.43] [994.43][S03]But because this is multimodal, because you have video,[997.52] [997.52][S03]because you have audio, because you have the text language[1002.01] [1002.01][S03]model running in the background, you[1004.17] [1004.17][S03]do have more levers to potentially pull here.[1007.022] [1007.022][S02] Yeah, yeah.[1007.98] [1007.98][S02]I think it should be able to resolve ambiguity[1011.79] [1011.79][S02]with more context.[1013.9] [1013.9][S03] How about different languages?[1016.305] [1016.305][S03]Is it only in English at the moment, and only[1018.18] [1018.18][S03]with a very clear accent?[1021.6] [1021.6][S02] It's mostly in English for me, but it's--[1024.63] [1024.63][S02]no, it's very multilingual.[1026.17] [1026.17][S02]So that's a function of being native audio.[1029.829] [1029.829][S02]It knows about 20 languages with pretty high proficiency.[1033.13] [1033.13][S02]And you can switch between languages[1035.49] [1035.49][S02]even in the same conversation.[1037.315] [1037.315][S03] So go on.[1038.19] [1038.19][S03]Give a little demo of different languages.[1040.01] [1040.01][S04] Hello, Gregory.[1041.26] [1041.26][S04]It's nice to speak with you again.[1044.099] [1044.099][S02] Bonjour, Astra.[1045.3] [1045.3][SPEAKING IN FRENCH][1047.31] [1065.84][S03] Oh.[1067.312] [1067.312][S03]"Little redhead."[1068.02] [1068.02][S03]Got that.[1068.66] [1068.66][S02] [SPEAKING FRENCH][1071.39] [1075.057][S03] So can you just-- hang on.[1076.64] [1076.64][S03]I did Russian in school.[1078.2] [1078.2][S03]I can remember one remaining Russian phrase.[1080.36] [1080.36][S03]Can you-- can you switch languages in the middle[1085.52] [1085.52][S03]without necessarily warning it?[1087.48] [1087.48][S03]So could I say, for example, [RUSSIAN]?[1089.81] [1092.135][S04] You certainly could use that phrase,[1094.26] [1094.26][S04]but what did you wish to ask about this phrase?[1097.17] [1097.17][S03] What does it mean?[1100.28] [1100.28][S04] [RUSSIAN] means "At what time does it open?"[1105.2] [1105.2][S04]It is asking about the opening time of something.[1110.53] [1110.53][S03] OK.[1111.7] [1111.7][S03]I mean, it is "What time does the shop open?"[1115.19] [1115.19][S03]I think.[1117.22] [1117.22][S03]But notable, then, that you're not saying,[1121.0] [1121.0][S03]now in English, now in French, now in Russian.[1123.555] [1123.555][S02] I think it's targeted[1124.93] [1124.93][S02]to respond in the language that you started with.[1127.19] [1127.19][S02]So actually, you said something in English[1129.19] [1129.19][S02]and then gave the Russian.[1130.7] [1130.7][S02]I think if you'd started speaking in Russian,[1132.14] [1132.14][S02]it would responded in Russian.[1133.39] [1133.39][S02]But as it was, it was thinking, I'm speaking English,[1137.39] [1137.39][S02]but I am hearing Russian.[1138.98] [1138.98][S02]So you didn't have to change that.[1140.48] [1140.48][S02]Bu, If you'd just started speaking in Russian,[1142.675] [1142.675][S02]it would have been maybe a little bit better.[1144.55] [1144.55][S03] I mean, this is different,[1146.133] [1146.133][S03]though, I mean, from the chat box that we have at the moment.[1148.705] [1148.705][S03]This is an additional capability.[1150.28] [1150.28][S02] I'm actually really excited about language[1152.53] [1152.53][S02]learning with this system, like, walking around[1154.488] [1154.488][S02]and being like, What is that? having it teach you the same way[1157.21] [1157.21][S02]that I was taught in school, where we would bring in objects[1161.59] [1161.59][S02]and talk about those objects in French class[1163.45] [1163.45][S02]to learn about stuff being together and learning language.[1168.09] [1168.09][S03] I can imagine being lost in a foreign city and that[1170.81] [1170.81][S03]being quite a helpful aid.[1172.167] [1172.167][S02] Exactly, yeah.[1173.25] [1173.25][S02]And it should be able to understand[1174.708] [1174.708][S02]other people speaking to you quite naturally, too, so--[1177.06] [1177.06][S03] So if this is the thing[1177.92] [1177.92][S03]that you're interacting with, what's[1179.78] [1179.78][S03]actually going on underneath the hood?[1182.01] [1182.01][S03]What are all the different components?[1183.6] [1183.6][S02] Yeah.[1184.308] [1184.308][S02]So the first thing is there's an app.[1187.04] [1187.04][S02]And that is actually gathering your video[1191.06] [1191.06][S02]and taking in your audio through the mic and so forth.[1195.32] [1195.32][S02]And that's connecting to a server[1197.75] [1197.75][S02]on which there are several different kinds[1200.18] [1200.18][S02]of neural network models.[1201.9] [1201.9][S03] Like what?[1203.315] [1203.315][S02] So there's a vision encoder and an audio encoder.[1207.36] [1207.36][S02]There's also specialized audio systems[1210.68] [1210.68][S02]that are just responsible for understanding when you've[1215.6] [1215.6][S02]probably stopped speaking.[1217.04] [1217.04][S02]Those are sitting next to the large language model, Gemini.[1222.62] [1222.62][S02]And they are sending information from these sensory encoders[1227.23] [1227.23][S02]directly into Gemini, which is responding.[1229.43] [1229.43][S02]We worked together with some of the teams in Gemini[1231.67] [1231.67][S02]also to change the Gemini model, to be better at dialogue[1235.75] [1235.75][S02]and audio processing.[1236.81] [1236.81][S02]So we've kind of improved its ability[1239.71] [1239.71][S02]to use, have audio, take an audio, and speak.[1243.19] [1243.19][S02]When we started working with the models,[1246.67] [1246.67][S02]they were making lots of factual errors.[1249.41] [1249.41][S02]So we had to identify ways in which we[1251.68] [1251.68][S02]could improve their factuality while[1255.52] [1255.52][S02]also being kind of conversational.[1258.58] [1258.58][S02]That was one aspect of our work on Gemini.[1261.14] [1261.14][S02]On top of all of that, though, is something called an agent.[1265.09] [1265.09][S02]The agent is taking the video and audio[1269.32] [1269.32][S02]and sending it to the model.[1271.13] [1271.13][S02]It's also calling search tools, so either Google Lens or Google[1277.69] [1277.69][S02]Search or Google Maps, when needed to respond to a query.[1284.01] [1284.01][S02]So if you ask about the price of something, it will call Search.[1288.81] [1288.81][S02]There's also a memory system that is being, in a sense,[1294.87] [1294.87][S02]part of the agent.[1297.21] [1297.21][S02]And offline in between sessions, the memory system[1300.93] [1300.93][S02]will summarize relevant information from the session,[1304.08] [1304.08][S02]about you and about what you've talked about in that session.[1308.29] [1308.29][S02]So those are some of the ingredients.[1310.33] [1310.33][S03] I mean, I'm trying to imagine something that we[1315.15] [1315.15][S03]use just to recognize a book.[1317.05] [1317.05][S03]I'm trying to think of the number of different elements[1319.59] [1319.59][S03]that are coming into play here, because you've got the computer[1322.74] [1322.74][S03]vision.[1323.47] [1323.47][S03]You've got the voice recognition.[1324.868] [1324.868][S03]You've got the large language models.[1326.41] [1326.41][S03]You've got the Google Search sort of sitting underneath it.[1329.23] [1329.23][S03]You've got the agent layer where you're actually[1331.71] [1331.71][S03]making decisions.[1332.73] [1332.73][S03]And you're doing all of that with almost no latency at all[1337.2] [1337.2][S03]in the answers that it's giving you.[1338.8] [1338.8][S03]I mean, this is a phenomenally complicated thing.[1341.02] [1341.02][S02] Yeah.[1341.22] [1341.22][S02]I mean, phenomenally complicated.[1342.84] [1342.84][S02]Of course, as engineers, we come up with abstraction layers[1345.81] [1345.81][S02]that we don't have to think about all the levels[1347.81] [1347.81][S02]of complexity at one time.[1349.94] [1349.94][S02]But I think overall, it's hugely complicated.[1352.8] [1352.8][S02]The data that's going into the models[1355.22] [1355.22][S02]is understood by very few people.[1357.36] [1357.36][S02]And exactly why it produces the results[1359.215] [1359.215][S02]is probably understood by no one,[1360.59] [1360.59][S02]in a sense, since it's just based on benchmarks.[1363.51] [1363.51][S03] Well, let me talk a little bit about the history[1364.79] [1364.79][S03]of this, because--[1365.75] [1365.75][S03]so back in the first series of this podcast,[1368.69] [1368.69][S03]you were a guest on the very first episode.[1370.92] [1370.92][S03]And then you were drawing on inspiration[1373.94] [1373.94][S03]from the animal kingdom for your research on intelligence.[1378.12] [1378.12][S03]And specifically, there was a bird, the western scrub jay,[1382.77] [1382.77][S03]that you were telling us about, as a way[1386.15] [1386.15][S03]to inspire more sophisticated memory for AI.[1388.825] [1388.825][S03]Let me just play you a little clip of it, actually.[1390.95] [1390.95][AUDIO PLAYBACK][1391.617] [1391.617][S03]- Having a kind of large database of things that[1395.24] [1395.24][S03]you've done and seen that you can access and that you can use[1399.68] [1399.68][S03]use to then guide your goal-directed behavior later--[1403.84] [1403.84][S03]I'm hungry.[1405.01] [1405.01][S03]Mm, I would love to have some maggots right now.[1407.29] [1407.29][S03]Where should I go find those?[1409.8] [1409.8][S03]That's the kind of thing we would like to replicate.[1412.847] [1412.847][END PLAYBACK][1413.43] [1413.43][S03] Have you managed to?[1414.763] [1414.763][S02] [LAUGHS][1417.358] [1418.97][S02]Hello, Project Astra.[1419.95] [1419.95][S02]Can you find some maggots for me?[1421.728] [1421.728][S03] I mean, that sounds quite[1423.27] [1423.27][S03]a lot like your orange juice example, doesn't it?[1425.35] [1425.35][S02] It is a proactive memory example, yeah.[1427.5] [1427.5][S03] Yeah.[1428.208] [1428.208][S03]And that's what you've done with Project Astra.[1431.292] [1431.292][S02] Yeah.[1432.0] [1432.0][S02]I think that there's a sense in which intelligence is really[1434.88] [1434.88][S02]one thing.[1436.12] [1436.12][S02]And one has a career and one is studying what intelligence is.[1441.79] [1441.79][S02]One is taking kind of glancing hits at it,[1443.95] [1443.95][S02]kind of trying to understand, spar with it in one way[1446.67] [1446.67][S02]or another.[1447.4] [1447.4][S02]And this project, maybe, is the strongest unification[1454.23] [1454.23][S02]of all of the strands of research I've had in my life.[1458.01] [1458.01][S02]Although, actually it's missing a major one,[1460.62] [1460.62][S02]which is that it's not embodied in a physical sense.[1462.87] [1462.87][S02]It can't act in the world--[1464.263] [1464.263][S03] Yet.[1464.93] [1464.93][S02] --which I--[1467.27] [1467.27][S02]perhaps, yeah.[1468.38] [1468.38][S02][LAUGHS] Yeah.[1470.73] [1470.73][S02]So yeah, I think memory, perception,[1475.003] [1475.003][S02]these have been long-standing interests.[1476.67] [1476.67][S02]And I think this is a way of bringing[1479.72] [1479.72][S02]them together that people seem to also find stimulates them.[1484.68] [1484.68][S02]They feel connected to it.[1486.66] [1486.66][S03] So how much of your neuroscience background[1489.2] [1489.2][S03]did end up inspiring Project Astra?[1491.2] [1491.2][S02] So neuroscience is used in two ways.[1493.2] [1493.2][S02]One is that there's a sense in which we're[1495.65] [1495.65][S02]using neuroscience to know when we've[1497.217] [1497.217][S02]done a good enough job to think about,[1498.8] [1498.8][S02]What does memory really mean?[1500.58] [1500.58][S02]and, Have we achieved it yet?[1503.15] [1503.15][S02]And it's also just a bit of a propulsion,[1505.97] [1505.97][S02]like, say, if we want something that is compatible with us,[1509.97] [1509.97][S02]human compatible, and, in some ways, like us, kind of[1514.04] [1514.04][S02]maybe go towards an embodiment of intelligence that's[1519.31] [1519.31][S02]a little bit more like us rather than a straightforward text[1522.79] [1522.79][S02]interface.[1523.52] [1523.52][S02]For example, I have been interested in the work[1526.78] [1526.78][S02]of Michael Tomasello, who studies[1530.5] [1530.5][S02]human communication by comparison to the great apes.[1536.89] [1536.89][S02]And he's really maybe the main thinker[1539.05] [1539.05][S02]behind this idea of, for me, situated dialogue,[1543.19] [1543.19][S02]where he talks about the basic premise of communication[1549.97] [1549.97][S02]as being about two individuals who are in the same place, who[1554.95] [1554.95][S02]are directing attention in the same place,[1558.17] [1558.17][S02]and therefore, inferring goals together[1561.28] [1561.28][S02]and then able to collaborate.[1563.32] [1563.32][S02]And that was kind of what we modeled in this technology.[1567.233] [1567.233][S03] So it's like the inspiration[1568.9] [1568.9][S03]rather than necessarily-- at the theoretical level,[1574.25] [1574.25][S03]rather than actually directly copying the design of it.[1577.11] [1577.11][S02] Not for the problem solving or the engineering[1579.76] [1579.76][S02]per se.[1580.26] [1580.26][S02]Then I think you need to come up with different solutions[1582.26] [1582.26][S02]that are dependent on the technology itself.[1584.37] [1584.37][S03] If Project Astra links to things[1587.87] [1587.87][S03]that we were talking about, I mean, literally years ago,[1590.58] [1590.58][S03]where did the first spark for this project come from?[1594.18] [1594.18][S03]Like, when did it actually begin?[1595.652] [1595.652][S02] Yeah.[1596.36] [1596.36][S02]So I think I know that--[1598.82] [1598.82][S02]Demis Hassabis, the CEO of DeepMind,[1604.43] [1604.43][S02]kind of threw down a challenge to the company[1607.52] [1607.52][S02]in a way, which was for us to think about what[1613.1] [1613.1][S02]a proto artificial general intelligence was,[1616.79] [1616.79][S02]which, what does that mean?[1618.89] [1618.89][S02]A proto artificial general intelligence[1621.53] [1621.53][S02]is a system that, if we created it and technically[1625.19] [1625.19][S02]minded people were able to scrutinize, investigate it, use[1628.07] [1628.07][S02]it, experience it, they would conclude[1630.59] [1630.59][S02]that the real deal, something that is generally intelligent[1636.19] [1636.19][S02]and a computational device, was ultimately going to arrive.[1641.15] [1641.15][S02]It was a matter of when, not if.[1643.03] [1643.03][S02]But that was left unspecified.[1644.51] [1644.51][S02]So there was a lot of creative thinking at the time like, well,[1648.372] [1648.372][S02]maybe it's this, maybe it's this, and so forth.[1650.33] [1650.33][S02]And some people had ideas of an intelligence arising[1656.11] [1656.11][S02]the same way that AlphaZero arose,[1658.21] [1658.21][S02]just by interacting with the world.[1659.96] [1659.96][S02]Other people maybe had other ideas,[1661.97] [1661.97][S02]but my idea was very much about the sociality of intelligence.[1666.08] [1666.08][S02]So we are not very smart as human beings[1669.67] [1669.67][S02]unless we learn from others or we[1671.29] [1671.29][S02]learn from books, which is the same as learning from others.[1673.79] [1673.79][S02]And that was the kind of idea I had for what proto AGI would be.[1677.81] [1677.81][S02]Then I thought also, we could unify[1679.51] [1679.51][S02]proto AGI with the idea of a helpful assistant whose[1683.23] [1683.23][S02]main goal is the benefit of the humans it interacts with.[1687.04] [1687.04][S02]So maybe those two things together[1688.78] [1688.78][S02]gave me something of a direction to look.[1693.07] [1693.07][S02]And then it was when I tried to think about making it ultimately[1697.02] [1697.02][S02]very natural, I sort of moved towards thinking about video[1700.95] [1700.95][S02]as the ultimately connective fiber of the systems.[1704.14] [1704.14][S03] Were there big moments along the way where you[1707.01] [1707.01][S03]had these big breakthroughs?[1708.678] [1708.678][S02] Where we had big breakthroughs?[1710.47] [1710.47][S02]Yeah, I think so.[1711.81] [1711.81][S02]There were these phases of the project.[1714.04] [1714.04][S02]So the first phase of the project[1715.56] [1715.56][S02]was basically a hackathon where we[1720.33] [1720.33][S02]had two weeks of making the first version.[1723.55] [1723.55][S02]And we have a video from that time.[1725.95] [1725.95][S02]It was quite crude.[1728.85] [1728.85][S02]But I remember Malcolm Reynolds, who's[1732.78] [1732.78][S02]a friend and engineer here, was playing around with Astra.[1736.72] [1736.72][S02]And he was going around an office room and saying,[1740.14] [1740.14][S02]what is this?[1740.95] [1740.95][S02]And the system would say, a plant.[1743.418] [1743.418][S02]And he'd say, what kind of a plant is this?[1745.21] [1745.21][S02]And it would say, a plant.[1746.293] [1746.293][LAUGHTER][1747.72] [1747.72][S02]Wasn't super flexible.[1749.26] [1749.26][S02]I remember the first demo I ever saw had a 7-second latency.[1753.72] [1753.72][S03] So you would say, Hi, Project Astra,[1755.998] [1755.998][S03]or whatever it was called then.[1757.29] [1757.29][S02] It wasn't called-- yeah.[1758.16] [1758.16][S03] And then 7 seconds later, it would--[1760.212] [1760.212][S02] Yeah.[1760.92] [1760.92][S02]So it was very difficult to use at all because you would kind of[1763.97] [1763.97][S02]think it had gone away, right?[1765.22] [1765.22][S02]But it was just 7 seconds later it would come back to you.[1767.705] [1767.705][LAUGHS][1769.56] [1769.56][S02]I think one of the main discoveries of the time[1771.99] [1771.99][S02]was basically that there's this idea of a prompt.[1775.36] [1775.36][S02]A prompt is the instructions you give to the system[1780.42] [1780.42][S02]that it needs for operation.[1781.9] [1781.9][S02]So systems like this really understand language.[1785.23] [1785.23][S02]They can read.[1785.94] [1785.94][S02]And you can say things to them like, your name is Astra.[1788.74] [1788.74][S02]You're an intelligent, helpful AI assistant.[1792.27] [1792.27][S02]Some of that information is inherent in the Gemini models[1795.33] [1795.33][S02]now.[1796.02] [1796.02][S02]But some of it's indicated in our prompt.[1798.91] [1798.91][S02]And it wasn't really understood whether we[1800.97] [1800.97][S02]could prompt a system that was multimodal before very well.[1804.97] [1804.97][S02]One of the things that was kind of a mind-blowing insight[1809.46] [1809.46][S02]or realization at the time was just telling the system[1812.49] [1812.49][S02]that it could see the world through the user's camera gave[1816.9] [1816.9][S02]it a sense of its own perspective[1818.64] [1818.64][S02]on things, like, where the provenance[1821.37] [1821.37][S02]of this information to it.[1823.09] [1823.09][S02]It didn't understand that before.[1824.47] [1824.47][S02]It was always making mistakes.[1826.01] [1826.01][S02]When you'd say, What do you see?[1827.56] [1827.56][S02]It was always giving me the wrong answer.[1829.268] [1829.268][S02]But then when we said, you're a system that[1831.39] [1831.39][S02]is an AI that is seeing through the user's camera,[1834.01] [1834.01][S02]then it could understand that this camera was[1837.27] [1837.27][S02]something it was effectively seeing[1838.95] [1838.95][S02]and it would answer correctly.[1839.915] [1839.915][S03] Wow.[1840.28] [1840.28][S02] I mean, there was a lot of work to do there,[1842.613] [1842.613][S02]but realizing that we could effectively prompt it was maybe[1845.233] [1845.233][S02]the-- even though it was a different kind of a system than[1847.65] [1847.65][S02]one we'd built before, and that you could use text to prompt its[1850.86] [1850.86][S02]understanding of the situated or more embodied--[1855.13] [1855.13][S03] That's so interesting.[1856.72] [1856.72][S03]When the gauntlet was thrown down of create a proto AGI,[1861.19] [1861.19][S03]were there people expressing doubt or skepticism[1864.27] [1864.27][S03]that something like this might have been possible?[1866.552] [1866.552][S02] Yeah.[1867.26] [1867.26][S02]So hindsight is curious in AI because it moves so fast,[1873.08] [1873.08][S02]and people's perception of what is obvious changes so fast.[1877.29] [1877.29][S02]I think it's obvious to a lot of people[1879.02] [1879.02][S02]now in some ways, which blows my mind.[1882.27] [1882.27][S02]I'm like, do you know the adversity, how much convincing[1886.76] [1886.76][S02]had to happen?[1888.947] [1888.947][S03] Well, tell us.[1890.03] [1890.03][S03]How much?[1891.06] [1891.06][S02] So I think from many different perspectives,[1893.94] [1893.94][S02]people thought this was an odd thing to do.[1897.41] [1897.41][S02]So from the perspective of, Could the systems actually[1901.73] [1901.73][S02]understand the world at all?[1905.0] [1905.0][S02]the vision systems in that era, in terms[1907.52] [1907.52][S02]of the number of pixels they were taking in,[1909.93] [1909.93][S02]it was like 96 by 96 patches of image.[1913.8] [1913.8][S02]So for those who don't know, the minimum of our screens is 1,000[1918.71] [1918.71][S02]pixels by 1,000 pixels--[1920.27] [1920.27][S02]so you know, a very blurry input to these systems.[1924.3] [1924.3][S02]So no wonder it couldn't identify what kind of a plant[1926.61] [1926.61][S02]it was.[1927.11] [1927.11][S02]It barely could see.[1928.48] [1928.48][S02]The fact that these systems would really know information[1933.46] [1933.46][S02]about what they're seeing, rather than just being[1936.37] [1936.37][S02]able to identify or classify what they're seeing, so having[1939.61] [1939.61][S02]a deep conversation about something,[1941.665] [1941.665][S02]seemed probably a little too far ahead.[1943.85] [1943.85][S02]We didn't even have basic knowledge of the amount of data[1947.32] [1947.32][S02]that you'd need to get for systems to perform[1949.51] [1949.51][S02]at various levels.[1950.72] [1950.72][S03] So then, I mean, OK.[1952.63] [1952.63][S03]If all of this seemed so absurd, and yet you[1956.29] [1956.29][S03]embarked on it anyway, were there[1958.25] [1958.25][S03]times when you thought it wasn't going to be possible?[1960.5] [1960.5][S02] No.[1961.125] [1961.125][S02]No, no, no, no.[1962.18] [1962.18][S02]It seemed always like it would be possible.[1966.43] [1966.43][S02]There were times that I was maybe willing to give up.[1968.872] [1968.872][S03] Oh, really?[1969.83] [1969.83][S02] Yeah.[1970.31] [1970.31][S02]I think it was--[1971.0] [1971.0][S02]I think there was a slow period before Gemini where things[1975.22] [1975.22][S02]weren't working very well.[1976.61] [1976.61][S02]And it was hard times.[1978.47] [1978.47][S02]I think it didn't seem like it was[1980.29] [1980.29][S02]a fruitful line of investigation at the time for some people.[1984.6] [1984.6][S02]But I sort of never wavered about the fact[1987.86] [1987.86][S02]that this was definitely possible.[1990.21] [1990.21][S02]I think I had a much more obstinate, stubborn,[1994.32] [1994.32][S02]and ultimately stupid way of going about it, which was just[1997.22] [1997.22][S02]like, if I work on this for long enough, it will definitely work.[1999.94] [1999.94][LAUGHS][2001.75] [2001.75][S03] So I heard that as part of the testing phase,[2004.25] [2004.25][S03]you have this Project Astra room.[2008.71] [2008.71][S03]What's going on in there?[2010.34] [2010.34][S03]What's in the room?[2011.33] [2011.33][S02] There is a special room, yeah.[2012.79] [2012.79][S03] What's inside the special room?[2014.582] [2014.582][S02] We have just all sorts of fun and games[2016.78] [2016.78][S02]in the special room.[2017.84] [2017.84][S02]There's a whole bar there so Astra can help you make a drink.[2022.46] [2022.46][S03] Right.[2023.21] [2023.21][S03]Yeah, it can.[2023.89] [2023.89][S02] There's an art gallery[2026.23] [2026.23][S02]so you can flash up different paintings on screens[2028.81] [2028.81][S02]and walk around the gallery and ask questions about art.[2031.725] [2031.725][S03] OK.[2032.35] [2032.35][S03]Well, let's dig into some of the stuff that's[2034.33] [2034.33][S03]going on behind the scenes of Astra a little bit more.[2036.64] [2036.64][S03]Latency, I think, is a really key thing.[2039.26] [2039.26][S03]You mentioned a moment ago about the 7-second lag that you used[2042.78] [2042.78][S03]to get.[2043.33] [2043.33][S03]How have you actually improved that?[2045.4] [2045.4][S02] So it's on multiple fronts.[2047.05] [2047.05][S02]So we've improved the actual streaming video.[2049.989] [2049.989][S02]So it actually is sending information faster[2053.76] [2053.76][S02]through the app.[2055.35] [2055.35][S02]There's a sense in which these systems, although they're[2058.469] [2058.469][S02]trained together, there's a vision[2060.36] [2060.36][S02]system and an audio system, this language model[2062.969] [2062.969][S02]system that is getting the information from those two[2065.5] [2065.5][S02]things.[2066.0] [2066.0][S02]What's called collocating them-- these[2068.04] [2068.04][S02]are kind of a technical term, but basically, we're[2070.44] [2070.44][S02]always processing images.[2072.28] [2072.28][S02]So as the video is coming in, for example,[2075.01] [2075.01][S02]into the vision system, it's always[2078.239] [2078.239][S02]running as fast as it can.[2080.08] [2080.08][S02]And then it's sitting in the same place,[2083.67] [2083.67][S02]in the same cluster of computers,[2086.159] [2086.159][S02]as is the large language model, so that it[2088.949] [2088.949][S02]doesn't have to make a call across a country or a continent.[2094.51] [2094.51][S03] Well, so--[2095.94] [2095.94][S03]sorry.[2096.88] [2096.88][S02] So they're running next to each other.[2097.77] [2097.77][S03] So to get this kind of real-time understanding[2100.187] [2100.187][S03]of what's going on, you have to physically locate[2103.25] [2103.25][S03]the computer hardware that are running these models[2105.59] [2105.59][S03]close to each other, because that makes a difference?[2108.102] [2108.102][S02] Absolutely, yeah.[2109.31] [2109.31][S03] Has that been the main thing, then,[2110.63] [2110.63][S03]just moving where you're actually running the models?[2113.01] [2113.01][S02] No.[2113.635] [2113.635][S02]So moving where we--[2115.97] [2115.97][S02]putting the models together is one thing.[2118.41] [2118.41][S02]Making sure that we are caching the context[2121.68] [2121.68][S02]so that the context of history, of what the system is[2124.94] [2124.94][S02]interacting with you, is incrementally updated over time.[2129.635] [2129.635][S02]There's this idea of doing work with native audio, which[2132.29] [2132.29][S02]means that previous systems had a text recognition[2138.62] [2138.62][S02]system-- or speech-to-text recognition system.[2141.24] [2141.24][S02]So they'd take in the audio, then they'd[2144.41] [2144.41][S02]produce a transcript, then they'd[2146.09] [2146.09][S02]call the language model, which would respond to that,[2148.41] [2148.41][S02]and then you'd get a response.[2150.12] [2150.12][S02]This system is directly getting the audio in,[2153.248] [2153.248][S02]so it doesn't have to have that secondary system, which[2155.54] [2155.54][S02]also takes time or produces extra latency.[2158.22] [2158.22][S02]So actually, there's a simple effect[2160.25] [2160.25][S02]that's possible with native audio is[2161.75] [2161.75][S02]it can understand rare words or the pronunciation of words.[2165.21] [2165.21][S02]A rare word, although becoming not so rare anymore,[2168.56] [2168.56][S02]or name is "Demis Hassabis."[2171.11] [2171.11][S02]The old systems that didn't understand audio natively[2174.56] [2174.56][S02]directly often thought I was saying "Damascus,"[2177.56] [2177.56][S02]but now it knows that it's Demis Hassabis.[2179.72] [2179.72][S02]And it can use context to resolve that.[2181.5] [2181.5][S02]The CEO of DeepMind is Demis Hassabis.[2184.04] [2184.04][S02]Another example that somebody found recently,[2186.12] [2186.12][S02]which we have a little demo of, is distinguishing between[2189.5] [2189.5][S02]the word "scahn" and the words "scohn,"[2191.7] [2191.7][S02]which are two pronunciations of the same biscuity thing.[2194.73] [2194.73][S02]Project Astra can actually-- you could say,[2197.587] [2197.587][S02]What's the difference between "scohn" and "scahn"?[2199.67] [2199.67][S02]It will have heard that you said a different word rather[2202.1] [2202.1][S02]than just transcribing it into the one word.[2204.36] [2204.36][S02]Then the final one is that team did a lot of great work[2206.87] [2206.87][S02]on what's called endpointing, which is a very technical term.[2210.06] [2210.06][S02]But more or less, it knows exactly when[2212.72] [2212.72][S02]you have stopped speaking.[2214.35] [2214.35][S02]So it's very good at sensing, OK,[2217.32] [2217.32][S02]the user is really done now so I can talk.[2219.24] [2219.24][S02]Then there's something even more sophisticated,[2221.198] [2221.198][S02]which is that it plans a response even if you[2223.31] [2223.31][S02]hadn't finished speaking yet.[2225.77] [2225.77][S02]And it sort of is speculatively planning that.[2228.45] [2228.45][S02]So it's sort of guessing, this is what I would say.[2231.83] [2231.83][S02]And then when it figures out that the user really[2235.01] [2235.01][S02]has finished speaking, then it just sends it right off.[2237.805] [2237.805][S02]So it's already done.[2238.68] [2238.68][S02]It's already figured out what to say before.[2240.513] [2240.513][S02]It's before maybe even you really[2242.84] [2242.84][S02]know that you're done speaking.[2244.17] [2244.17][S03] That is so interesting[2245.54] [2245.54][S03]because I guess, actually, a lot of the time,[2247.26] [2247.26][S03]people's sentences, the important bit[2248.802] [2248.802][S03]of their sentences, can be in the middle[2251.9] [2251.9][S03]and then they sort of trail off towards the end.[2254.52] [2254.52][S03]And then you can use that time for getting ready[2257.6] [2257.6][S03]with your answer.[2258.432] [2258.432][S02] Yeah, pretty much that.[2259.89] [2259.89][S02]Yeah.[2260.39] [2260.39][LAUGHTER][2262.43] [2262.43][S02]Oh, yeah.[2262.95] [2262.95][S02]We talked about this stuff--[2264.41] [2264.41][S02]actually, we talked about that stuff three years ago.[2267.9] [2267.9][S02]And then it seemed like, oh, that's too much.[2270.21] [2270.21][S02]And then it kind of started to work this year.[2272.45] [2272.45][S03] Preemptively guessing[2273.89] [2273.89][S03]what the answer is going to be before the conversation has[2276.5] [2276.5][S03]got to that point.[2277.35] [2277.35][S02] Yeah.[2278.058] [2278.058][S02]And it's hard.[2278.742] [2278.742][S02]We pause for a long time in our sentences.[2283.015] [2283.015][S02]So the system that we have actually has to use some, quote,[2287.46] [2287.46][S02]"semantic understanding," since it also has a bit[2290.84] [2290.84][S02]of understanding of context and the sounds.[2294.05] [2294.05][S02]It's also hearing to guess when the user's probably done.[2297.473] [2297.473][S03] But also, the reasoning that it's doing,[2299.64] [2299.64][S03]I mean, even separate from reasoning[2301.243] [2301.243][S03]whether it's finished a sentence or not.[2302.91] [2302.91][S03]Do you think that Project Astra is capable of reasoning?[2307.04] [2307.04][S02] Yeah.[2308.1] [2308.1][S02]It's primarily reasoning through its internal structure[2313.04] [2313.04][S02]inside the neural network, sort of in an unobservable--[2316.07] [2316.07][S02]or sort of a very complex way.[2318.89] [2318.89][S02]Then there's the dialogue itself it's producing.[2321.95] [2321.95][S02]So it sometimes reasons through the dialogue.[2324.18] [2324.18][S02]So you can hear it sounding out an answer.[2327.44] [2327.44][S02]People are also developing systems[2329.12] [2329.12][S02]that have inner speech, effectively,[2331.365] [2331.365][S02]where they're talking to themselves without talking[2333.49] [2333.49][S02]to you.[2333.98] [2333.98][S02]Project Astra, at the moment, doesn't do much of that.[2336.17] [2336.17][S03] But then I guess the advances that[2338.087] [2338.087][S03]happen in the reasoning models need not be distinct from what[2343.96] [2343.96][S03]happens in Project Astra, I guess,[2345.443] [2345.443][S03]is the whole point of this, is that it's[2347.11] [2347.11][S03]pulling in everything so that you have this ultimate proto[2350.74] [2350.74][S03]AGI, as you call it.[2351.98] [2351.98][S02] Yeah.[2352.688] [2352.688][S02]And in some ways, I actually hope[2354.46] [2354.46][S02]that it motivates some maybe more vigorous work[2359.41] [2359.41][S02]on some aspects of reasoning.[2361.07] [2361.07][S02]So we have this great example of Bibo Xu, the product[2364.45] [2364.45][S02]manager at Project Astra, pulled out Astra one day at lunch[2367.84] [2367.84][S02]and was like, how many calories are on my plate?[2370.4] [2370.4][S02]And she had a very complex, very beautifully laid out plate[2373.6] [2373.6][S02]with six types of food, some almonds in the middle,[2377.98] [2377.98][S02]a pork loin over there, some Brussels sprouts, or whatever.[2382.61] [2382.61][S02]And it was like, oh, you know, kind of waffled a little bit.[2386.63] [2386.63][S02]But then she said, keep a running total.[2389.5] [2389.5][S02]How many are in these Brussels sprouts?[2391.842] [2391.842][S02]And it was like, well, that's seven Brussels sprouts,[2394.05] [2394.05][S02]therefore it's this many calories.[2396.15] [2396.15][S02]And then, OK, now add the pork loin.[2398.453] [2398.453][S02]One of the things that was quite notable to me[2400.37] [2400.37][S02]was that Bibo was hand-holding its thinking.[2404.8] [2404.8][S02]As you said, it needs little guidance sometimes.[2406.8] [2406.8][S02]But I don't think that we're very far off from a system[2409.19] [2409.19][S02]that itself would just say, well,[2411.45] [2411.45][S02]I see there are seven almonds over there,[2413.64] [2413.64][S02]this many Brussels sprouts, there's a pork loin,[2416.04] [2416.04][S02]all those together are such and such.[2418.11] [2418.11][S02]So I think in some sense, it's not good at that stuff,[2421.47] [2421.47][S02]because we just haven't ever tried[2423.582] [2423.582][S02]to build a system that could reason about that stuff.[2425.79] [2425.79][S03] Now, I want to talk to you a bit more about memory.[2428.415] [2428.415][S03]On that point, about the things that are recalling and keeping[2433.01] [2433.01][S03]in its mind, as it were, if you'll[2434.6] [2434.6][S03]forgive the anthropomorphism, I know that back at Google I/O,[2438.68] [2438.68][S03]this could remember what had happened in the last 45 seconds.[2442.32] [2442.32][S03]And now you've increased that time.[2445.5] [2445.5][S03]You can do 10 minutes now, right?[2447.18] [2447.18][S02] Yeah, it's about 10 minutes.[2448.26] [2448.26][S02]Yeah, actually it's a little bit longer in some ways,[2450.468] [2450.468][S02]but 10 minutes is maybe what it should do on the 10.[2454.15] [2454.15][S03] What makes 10 minutes the limit?[2456.132] [2456.132][S02] Yeah.[2456.84] [2456.84][S02]So it's got, basically, a raw record of the last 10 minutes[2463.71] [2463.71][S02]of video.[2464.5] [2464.5][S02]It works at about one frame per second.[2466.33] [2466.33][S02]So it's got, basically, a stack of all the frames over time[2471.3] [2471.3][S02]and all the audio that came in between those frames, so[2474.36] [2474.36][S02]the last 600 frames or something like that.[2477.01] [2477.01][S02]The limits are really about the memory on the chips, I think.[2480.78] [2480.78][S02]That hasn't scaled very much, I think,[2482.64] [2482.64][S02]in the last decade or something like that,[2484.42] [2484.42][S02]the amount of this sort of fast, active memory.[2486.985] [2486.985][S03] But so at the moment, then,[2488.61] [2488.61][S03]it is effectively acting like a video recorder, as it were,[2491.11] [2491.11][S03]keeping an actual record of everything that's happened[2493.11] [2493.11][S03]in the previous 10 minutes.[2494.38] [2494.38][S02] Yeah.[2494.79] [2494.79][S02]Yeah, I mean, it's quite active.[2496.21] [2496.21][S02]It's able to use that information right away.[2498.25] [2498.25][S02]There's also a sort of a secondary system, which[2500.76] [2500.76][S02]is when you turn the system off, it[2504.54] [2504.54][S02]will then take that conversation and summarize it and pull out[2507.98] [2507.98][S02]relevant facts.[2509.22] [2509.22][S03] The most important bits.[2510.72] [2510.72][S02] Yeah.[2511.428] [2511.428][S02]And it uses its own discretion to figure out what that is.[2514.115] [2514.115][S03] To extract the gist of it, as it were.[2516.51] [2516.51][S02] Yeah.[2516.895] [2516.895][S03] But at the moment, I mean, one thing that it can do[2519.52] [2519.52][S03]is recall important things from recent interactions, can it?[2522.27] [2522.27][S02] Yeah.[2522.978] [2522.978][S02]It's kind of got sort of a two-stream memory.[2525.9] [2525.9][S02]So it's got a memory that is both about you as a person.[2530.01] [2530.01][S02]It's got a developing understanding of you.[2535.62] [2535.62][S02]It's effectively taking notes like, oh, they like ice cream,[2539.4] [2539.4][S02]that's chocolate ice cream, or whatever.[2541.1] [2541.1][S02]That'll be like a list of things it's discovered about you.[2544.98] [2544.98][S02]And that's actually updated after every session, too.[2548.49] [2548.49][S02]So suppose you say, you know what, I actually[2551.72] [2551.72][S02]decided that I don't like ice cream anymore.[2553.89] [2553.89][S02]I really like cake.[2554.94] [2554.94][S02]So forget that I liked ice cream.[2557.42] [2557.42][S02]It will then say, user says they no longer like ice cream,[2561.84] [2561.84][S02]they like cake.[2562.8] [2562.8][S02]And those things are kind of a stationary or static[2565.88] [2565.88][S02]understanding of who you are, effectively, or what you like,[2569.86] [2569.86][S02]your preferences.[2571.04] [2571.04][S02]Then there's also this kind of conversational summary[2574.39] [2574.39][S02]that's like, on Tuesday at 8:50, we talked about this game[2579.67] [2579.67][S02]of chess.[2580.993] [2580.993][S03] But then how does it decide which bit goes in which?[2583.66] [2583.66][S03]How does it decide what's important enough[2585.46] [2585.46][S03]to be a thing about you that it remembers?[2589.46] [2589.46][S02] So it's got heuristics.[2591.02] [2591.02][S02]These systems actually are given heuristics.[2593.03] [2593.03][S02]So a heuristic is basically a rule[2596.56] [2596.56][S02]of thumb for what to remember.[2598.97] [2598.97][S02]So one heuristic it uses is, and we've told it to,[2603.31] [2603.31][S02]is if you ask it to remember something it should definitely[2605.89] [2605.89][S02]remember that.[2607.27] [2607.27][S02]It's a pretty clear one.[2608.57] [2608.57][S02]So if I say, remember my door code,[2612.43] [2612.43][S02]it will do that because it will understand[2615.16] [2615.16][S02]that's an instruction of relevance.[2617.38] [2617.38][S02]Otherwise, it takes a best guess.[2620.05] [2620.05][S02]It tries to say--[2621.655] [2621.655][S02]it's sort of saying, has the user expressed any preferences[2624.75] [2624.75][S02]that are interesting or that are different from the ones[2627.39] [2627.39][S02]that the user has already expressed?[2629.473] [2629.473][S02]And then it will kind of update based on that.[2631.39] [2631.39][S03] Well, let's talk about some[2632.22] [2632.22][S03]of the privacy concerns here, then.[2633.76] [2633.76][S03]How do you mitigate against some of those privacy concerns?[2636.95] [2636.95][S02] Right.[2637.7] [2637.7][S02]So I think that one of the major standards is that of consent.[2642.01] [2642.01][S02]The users have access to their previously recorded data.[2647.49] [2647.49][S02]And they can delete it or see what is stored.[2651.58] [2651.58][S02]Every time you delete something, it[2653.43] [2653.43][S02]reconstitutes its whole knowledge of you.[2656.055] [2656.055][S03] Oh.[2656.68] [2656.68][S02] It goes through the whole process of summarizing[2659.22] [2659.22][S02]things it knows about you anew.[2660.895] [2660.895][S03] So the answer then, I[2662.27] [2662.27][S03]guess, is that the user ends up having some control over what[2666.18] [2666.18][S03]it knows about it--[2667.71] [2667.71][S03]about them, rather.[2668.62] [2668.62][S02] Yeah.[2669.328] [2669.328][S03] But actually, so in this podcast a few episodes ago,[2672.76] [2672.76][S03]we got to talk to Iason Gabriel, who's this ethicist at DeepMind.[2678.06] [2678.06][S03]He's amazing.[2679.44] [2679.44][S03]And he was telling us about the ethics of AI assistants[2683.38] [2683.38][S03]and how they should be shaped in order to take[2685.9] [2685.9][S03]into account lots of these difficult questions.[2688.64] [2688.64][S03]How much has his work fed into what[2692.38] [2692.38][S03]you've come up with with Astra?[2693.86] [2693.86][S02] We just fed his 243-page report into Astra.[2697.19] [2697.19][S02]And Astra said, OK, I got it.[2699.23] [2699.23][S03] Did you?[2700.224] [2700.224][S02] No, I wish.[2701.02] [2701.02][S03] Oh, that would have been so--[2702.728] [2702.728][S02] [LAUGHS] Yeah, I think[2704.74] [2704.74][S02]we've spoken a lot with Iason.[2706.7] [2706.7][S02]And we've done a lot of work with a team that he's part of.[2710.53] [2710.53][S02]And they've been investigating both the model and the agent[2715.12] [2715.12][S02]as a whole, exploring what it might[2717.67] [2717.67][S02]do in different circumstances, also working[2719.83] [2719.83][S02]with some external red teamers who[2721.6] [2721.6][S02]maybe have fewer preconceptions and might[2724.12] [2724.12][S02]do more different kinds of adversarial attacks[2727.51] [2727.51][S02]on the system.[2728.27] [2728.27][S02]We also have a layer of safety filters.[2730.67] [2730.67][S02]And this is for user harms or, for example,[2735.05] [2735.05][S02]if you say certain things to it or show it pornography,[2738.86] [2738.86][S02]for example, it will trigger these filters[2742.35] [2742.35][S02]and not respond to that.[2744.43] [2744.43][S02]It'll also trigger on its own speech,[2746.56] [2746.56][S02]so it can't say certain things, although they trigger[2749.52] [2749.52][S02]very infrequently anyway.[2750.682] [2750.682][S02]But I don't know.[2751.39] [2751.39][S02]Yeah, I think that the range of issues is quite broad.[2755.31] [2755.31][S02]Fortunately, we still have some time to figure stuff out.[2758.955] [2758.955][S03] OK.[2759.58] [2759.58][S03]So what, then, are your next priorities, then?[2761.43] [2761.43][S03]Over the next few months, what were the main things[2763.38] [2763.38][S03]you're going to be working on?[2764.35] [2764.35][S02] I'm very interested in something[2766.183] [2766.183][S02]called proactive video work.[2769.93] [2769.93][S02]So that is to say, a system that can not only[2774.48] [2774.48][S02]respond when you speak, but can also[2776.52] [2776.52][S02]help you in an ongoing sense.[2778.33] [2778.33][S02]So for example, that's part of the visual interpreter[2780.66] [2780.66][S02]for the blind problem.[2781.72] [2781.72][S02]So you're walking around, you can't see,[2783.84] [2783.84][S02]it will say, oh, watch out for the table over there.[2787.28] [2787.28][S02]It can guide you in an ongoing sense.[2789.78] [2789.78][S02]We're also doing a lot of work on more audio output, what's[2793.77] [2793.77][S02]called full duplex.[2795.04] [2795.04][S02]So it will process both--[2798.155] [2798.155][S02]it'll hear and speak at the same time, which[2801.13] [2801.13][S02]could be potentially annoying.[2802.38] [2802.38][S02]It could interrupt you.[2805.05] [2805.05][S02]But it's also more natural conversation.[2807.47] [2807.47][S02]As you're talking I might say, uh-huh, uh-huh.[2810.86] [2810.86][S02]And that's listening and talking at the same time.[2813.8] [2813.8][S02]It's part of language to confirm.[2815.87] [2815.87][S02]More on reasoning, as you said, more deep kinds of memory,[2821.61] [2821.61][S02]reflection of certain kinds.[2824.54] [2824.54][S02]When it calls tools to be able to do deeper inquiries[2829.16] [2829.16][S02]and research with tools, yeah, there's just so[2832.67] [2832.67][S02]many things to do better.[2834.24] [2834.24][S03] Well, thank you very much for joining us, Greg.[2835.98] [2835.98][S02] Thank you, Hannah.[2837.23] [2837.23][S03] It is strange how quickly our expectations change[2841.28] [2841.28][S03]about AI.[2842.155] [2842.155][S03]I don't know if you remember what[2843.53] [2843.53][S03]Oriol said in our last episode.[2845.1] [2845.1][S03]He said, if someone had told him five years ago the things that[2849.32] [2849.32][S03]would be possible, he would think that we were already[2852.14] [2852.14][S03]on the path to AGI.[2854.42] [2854.42][S03]And yet, here we have this prototype of a multimodal agent.[2859.33] [2859.33][S03]It's one that can see, that can hear,[2861.58] [2861.58][S03]that has memory and context and reasoning[2864.39] [2864.39][S03]and multilingual, real-time conversation.[2867.61] [2867.61][S03]This is an agent that could, at least in theory, accompany you[2871.98] [2871.98][S03]on your day to day, enhancing your knowledge,[2874.93] [2874.93][S03]supporting people with disabilities,[2876.91] [2876.91][S03]and augmenting our skills.[2879.25] [2879.25][S03]Now, of course, AGI, it isn't.[2882.45] [2882.45][S03]But it definitely feels like we are a significant leap[2886.203] [2886.203][S03]from the kinds of systems that we were talking[2888.12] [2888.12][S03]about even two years ago.[2891.435] [2891.435][S03]Thank you so much for joining us for this series of "Google[2893.96] [2893.96][S03]DeepMind--[2894.46] [2894.46][S03]The Podcast."[2895.15] [2895.15][S03]We are going to take a break from here,[2897.19] [2897.19][S03]but if you want to catch up on any of our previous episodes,[2900.61] [2900.61][S03]then there is a whole array of deliciously nerdy,[2904.27] [2904.27][S03]AI conversational delights in our back catalog for you[2907.95] [2907.95][S03]to enjoy.[2908.74] [2908.74][S03]Just find them on YouTube or wherever you get your podcasts.[2911.82] [2911.82][MUSIC PLAYING][2914.87]