[0.0][S08] What can you see right now, Gemini?[2.33] [2.33][S03] OK.[3.12] [3.12][S03]I see a woman with blonde hair sitting[4.85] [4.85][S03]at a desk with a microphone in front of her.[7.11] [7.11][S03]She has her arms crossed and is smiling.[9.45] [9.45][S03]There's a computer keyboard and mouse on the desk[11.78] [11.78][S03]and a lamp in the background.[13.457] [13.457][S03]What else can you tell me?[14.54] [14.54][S08] That has been so useful in diagnosing[17.57] [17.57][S08]my sick houseplants, I can't tell you.[19.47] [19.47][S09] And so if I describe[22.82] [22.82][S09]that I want to hear a song that reminds me[26.3] [26.3][S09]of being in a rainy cafe in Japan at nighttime,[29.97] [29.97][S09]that's something that you would really[32.6] [32.6][S09]want to have all of these different modalities coming[35.69] [35.69][S09]together to understand, as opposed to just guessing[41.63] [41.63][S09]at what a rainy cafe might sound like.[43.512] [43.512][S08] It's like a two-dimensional version[45.47] [45.47][S08]of a fully immersive experience.[46.893] [46.893][S09] Exactly.[47.81] [47.81][S08] And this is the notoriously difficult challenge[50.18] [50.18][S08]that people have used to see how good--[51.78] [51.78][S09] So instead of having one person eating[53.6] [53.6][S09]spaghetti, we just had 10 people eating spaghetti[55.85] [55.85][S09]all at the same table.[57.32] [57.32][S08] I mean, there was a lot of Will Smith eating[58.77] [58.77][S08]spaghetti for a number of years, I seem to remember.[61.06] [61.06][S09] Yes.[61.81] [61.81][S09]And so Will Smith eating spaghetti, that is so last year.[67.57] [67.57][S09]Let's try multiple people eating spaghetti.[69.88] [69.88][MUSIC PLAYING][73.044] [74.4][S08] Welcome to "Google DeepMind, The Podcast."[76.65] [76.65][S08]I'm Professor Hannah Fry.[77.833] [77.833][S08]Now, one of the things that we've always[79.5] [79.5][S08]done with this podcast is to bring you[81.93] [81.93][S08]access to the people who are working on some of the biggest[85.05] [85.05][S08]breakthroughs in AI.[86.44] [86.44][S08]And a lot of the time, the researchers,[88.45] [88.45][S08]they're talking about techniques and technology[90.81] [90.81][S08]that underpins big ideas.[93.64] [93.64][S08]But now, we are at a stage where more and more[96.81] [96.81][S08]of the tools that we have seen the early iterations of here[100.65] [100.65][S08]are now live.[102.1] [102.1][S08]They are out there in the world for you to interact with.[105.13] [105.13][S08]So what we wanted to do in this episode[106.98] [106.98][S08]is just to pause, to look at the array of tools[110.07] [110.07][S08]that have been released, talk about how they have changed[113.1] [113.1][S08]since we first encountered them, and to explore the myriad ways[116.7] [116.7][S08]that they can be used.[118.23] [118.23][S08]And if that is our objective for today,[120.22] [120.22][S08]well, there is no one better to show[121.98] [121.98][S08]us this progress than Paige Bailey, AI Developer Relations[126.06] [126.06][S08]Engineering Lead at Google DeepMind.[128.56] [128.56][S08]Paige, welcome to the podcast.[130.03] [130.03][S09] Thank you so much for having me.[131.97] [131.97][S08] The thing is is that we[133.23] [133.23][S08]get to see a lot of the early iterations of this stuff.[135.57] [135.57][S09] Yes.[136.32] [136.32][S08] And last year, we had Doug Eck on the show.[139.42] [139.42][S08]And he was showing us, I think, the very first iteration of Veo,[144.12] [144.12][S08]which now, with the launch of Veo 3, I mean,[147.03] [147.03][S08]it's quite a different beast.[148.843] [148.843][S09] Exactly.[149.76] [149.76][S09]The first implementation of the Veo model[152.16] [152.16][S09]was still just visual only, not including[156.48] [156.48][S09]all of the really enriching sound qualities that we[159.21] [159.21][S09]see from the Veo 3 model.[160.98] [160.98][S09]And you also had to give it pretty significant guidance[163.333] [163.333][S09]in order to get the model to produce something that looked[165.75] [165.75][S09]photorealistic, or even something that you[167.64] [167.64][S09]might see in a cinematic film.[169.41] [169.41][S09]But we've come a long way.[171.018] [171.018][S09]I would be really curious to see what[172.56] [172.56][S09]that first video looked like.[173.74] [173.74][S08] Yeah, so I think we have it, actually.[175.29] [175.29][S09] Yeah, let's do it.[176.69] [176.69][S08] We have this car racing through the streets.[179.88] [179.88][S08]You can see the neon lights reflected in the wet pavement[183.8] [183.8][S08]below.[184.56] [184.56][S08]There's other cars jostling for position around,[187.38] [187.38][S08]and it's almost like everything is blurred, because you're just[190.49] [190.49][S08]going so fast.[192.27] [192.27][S08]But it's really consistent.[193.53] [193.53][S08]Now, it's gone through a tunnel.[195.03] [195.03][S08]There are these big lights overhead,[197.97] [197.97][S08]and it's come out of the tunnel into an extremely realistic[203.06] [203.06][S08]modern scene.[203.608] [203.608][S02] It's a wow moment.[204.9] [204.9][S08] That's incredible.[205.77] [205.77][S02] That's a wow moment.[206.61] [206.61][S09] That is incredible.[208.02] [208.02][S08] I mean, that is really good, isn't it?[209.64] [209.64][S09] It is so good.[210.6] [210.6][S08] So some things that I[211.975] [211.975][S08]notice now, it is quite blurry.[214.86] [214.86][S08]And I guess that's part of the vibe that it's going for.[218.28] [218.28][S08]But you're not seeing this pristine detail on the car.[220.748] [220.748][S09] Not at all.[221.79] [221.79][S09]I think if we also looked really closely,[223.62] [223.62][S09]we might also see that some of the physics that's expressing[226.76] [226.76][S09]the shots is not quite right.[229.14] [229.14][S09]And the way that the light gets reflected on things[231.83] [231.83][S09]is also not necessarily quite consistent.[234.92] [234.92][S09]So let's see how well the new Veo 3 model does for this.[239.43] [239.43][S09]Yeah.[239.99] [239.99][S08] I mean, Doug was demonstrating the real limits[243.29] [243.29][S08]of what Veo was able to do.[245.245] [245.245][S08]And I suppose it's no coincidence[246.62] [246.62][S08]that it was a car that was racing through a scene.[249.66] [249.66][S08]If you tried to test Veo on, I don't know,[252.62] [252.62][S08]like a human face that was moving,[255.5] [255.5][S08]conveying a particular emotion, it might struggle a bit more.[258.12] [258.12][S09] Definitely.[259.279] [259.279][S09]It's very famous, the videos of people eating spaghetti,[263.75] [263.75][S09]or some of these things that are very,[266.52] [266.52][S09]very clearly not human characters[269.33] [269.33][S09]or not real life characters.[271.67] [271.67][S09]So it might be interesting also to just try with another prompt,[274.91] [274.91][S09]too, that is a little bit more human-centric,[277.82] [277.82][S09]but let's try with this.[278.82] [278.82][S08] Let's try this one.[279.62] [279.62][S08]So you're using exactly the same prompt here.[281.3] [281.3][S09] Exactly the same prompt.[282.883] [282.883][S09]And here we're in the Gemini app.[284.88] [284.88][S09]So you can see the exact same prompt generating this video.[288.36] [288.36][S09]It can take up to two to three minutes[290.39] [290.39][S09]in order to get it to have the right outputs.[293.29] [293.29][S08] One thing that was really noticeable[295.29] [295.29][S08]was the way that Doug's prompt was very poetic.[297.913] [297.913][S09] It was lovely.[299.08] [299.08][S08] It was gorgeous.[300.33] [300.33][S08]It was a little mini movie in its own right.[302.775] [302.775][S09] Yes.[303.525] [303.525][S08] What are your tips for generating these prompts?[305.71] [305.71][S09] So interestingly, we[307.23] [307.23][S09]have a new feature with the Veo 3 model called prompt rewriting,[311.62] [311.62][S09]which gives you the ability to give your input[314.49] [314.49][S09]sentence to the API, the way that we interact with the model,[318.87] [318.87][S09]and to get a response back that makes the prompt a lot more[322.23] [322.23][S09]detailed, a lot more aligned with perhaps[325.2] [325.2][S09]what you're imagining, so you don't necessarily[327.66] [327.66][S09]have to go through all of the mental work[330.9] [330.9][S09]and all of the terminology that would[332.97] [332.97][S09]be appropriate to describe this thing that you just imagined.[337.8] [337.8][S09]And I often use Gemini for something quite similar[341.28] [341.28][S09]is you can give Gemini the prompt that you're thinking[344.52] [344.52][S09]in your objective, and then ask it[346.59] [346.59][S09]to craft a prompt for a large language[348.9] [348.9][S09]model, or a video generation model,[351.4] [351.4][S09]in a way that would make the prompt much more[354.64] [354.64][S09]likely to produce the optimal output that you were expecting.[357.715] [357.715][S08] That's quite a good tip then, actually.[359.84] [359.84][S08]If you are not that good at writing prompts,[362.39] [362.39][S08]get Gemini to write the prompt better.[364.52] [364.52][S09] Yeah.[365.472] [365.472][S09]And so you can do that, kind of just[368.56] [368.56][S09]invoke it naturally through the Gemini app, which is probably[371.86] [371.86][S09]easiest for people to try.[373.6] [373.6][S09]But we also have the prompt rewriter hyper[376.84] [376.84][S09]specified for Veo 3 available through the API,[380.1] [380.1][S09]if people are a little bit more comfortable with programming[382.6] [382.6][S09]languages.[383.54] [383.54][S08] So how long are the Veo clips?[385.67] [385.67][S09] The Veo clips for the ones[387.4] [387.4][S09]that we released publicly, so the ones that people can try,[390.98] [390.98][S09]they're all around eight seconds in size, so very short.[394.01] [394.01][S08] Why does it need to be eight seconds long?[396.26] [396.26][S09] So it's eight seconds made available publicly.[399.73] [399.73][S09]Internally, we have models that are[401.44] [401.44][S09]capable of producing much more long-form content,[404.68] [404.68][S09]but we find that eight seconds is really good[407.23] [407.23][S09]to give you full creative control over that first clip.[411.52] [411.52][S09]And for the eight seconds, it's also useful in the sense[416.14] [416.14][S09]that you can get an idea of the style,[418.37] [418.37][S09]you can start experimenting with language.[420.44] [420.44][S09]And it's also wonderful in the sense[422.38] [422.38][S09]that you can start putting to life things[425.77] [425.77][S09]that you might have been imagining before.[428.14] [428.14][S09]I know that all of the internet is very enchanted with memes,[433.34] [433.34][S09]and now you can have memes that are much more long form,[436.49] [436.49][S09]that are actual videos, as opposed[438.76] [438.76][S09]to just the single snapshots.[441.522] [441.522][S08] Absolutely.[442.48] [442.48][S08]OK, it's ready.[443.105] [443.105][S09] Amazing.[444.022] [444.022][S09]Let's do it.[444.71] [444.71][S09]Oh, my gosh.[445.52] [445.52][S09]This is already--[446.47] [446.47][S08] All of a sudden.[447.32] [447.32][S09] This is so cool.[448.64] [448.64][S08] OK, now-- oh, wow.[450.05] [450.05][S08]Goodness me.[452.5] [452.5][S08]OK, it's like suddenly someone has switched on HD.[456.58] [456.58][S09] It absolutely is.[457.88] [457.88][S08] Especially that first shot.[460.34] [460.34][S08]So what we're looking at here, let me watch it again.[465.47] [465.47][S08]It looks like you're in "Blade Runner."[467.395] [467.395][S09] It does look like--[468.77] [468.77][S08] It's really evocative of that.[470.52] [470.52][S08]So the neon lights have changed from these sort of pink,[473.58] [473.58][S08]sort of horrifying neons to full on billboards, which you can't[477.62] [477.62][S08]totally see the detail of as you go past them,[479.97] [479.97][S08]but there's structure to them that it's included.[484.94] [484.94][S08]The car is now sort of painted with light, almost,[492.05] [492.05][S08]running through the scene.[494.1] [494.1][S08]But also look at the lighting on that bonnet.[496.14] [496.14][S08]Oh, my gosh, that's extraordinary.[498.142] [498.142][S09] It is gorgeous.[499.35] [499.35][S09]And the level of detail in all of the buildings.[502.55] [502.55][S09]Actually, this really does look like a city in a futuristic Hong[506.99] [506.99][S09]Kong.[507.63] [507.63][S08] What I'm noticing here is there's a spotlight.[510.09] [510.09][S08]A car comes out of the tunnel, there's a lamppost above it,[514.35] [514.35][S08]and the spotlight perfectly tracks[516.35] [516.35][S08]where you would expect it to be along the bonnet of the car.[518.85] [518.85][S09] That's amazing.[520.058] [520.058][S09]Do you hear the sound?[521.184] [521.184][S08] The screeching tires.[522.559] [522.559][S09] Yes.[524.268] [524.268][S08] Oh, sirens in the background.[525.976] [525.976][GASPS][526.85] [526.85][S09] This is so cool.[528.633] [528.633][S08] Oh, it's very evocative, isn't it?[530.55] [530.55][S09] It is.[532.91] [532.91][S08] Oh, yeah.[533.94] [533.94][S09] Yeah.[534.18] [534.18][S08] But the sound perfectly[535.31] [535.31][S08]matches the frames, then.[536.52] [536.52][S09] It definitely does.[537.895] [537.895][S09]And the background tracks the sounds[539.81] [539.81][S09]for these videos that get generated with Veo 3.[542.695] [545.85][S09]They're actually able to match not just things[549.68] [549.68][S09]like the cars, as they're kind of coming into the scene,[555.17] [555.17][S09]but also capable of giving you audio.[558.03] [558.03][S09]So audio is outputs, background music,[561.12] [561.12][S09]if you wanted to have cinematic music coupled with the audio[565.07] [565.07][S09]or coupled with the background noises, all[567.65] [567.65][S09]of this kind of stitched together into a single video.[570.39] [570.39][S09]It's pretty magical.[571.563] [571.563][S08] Let's try one of the hard ones, the spaghetti thing.[574.23] [574.23][S09] OK, so let's try a new video output.[578.19] [578.19][S08] Yeah.[578.9] [578.9][S09] What kind of spaghetti eating[581.243] [581.243][S09]would you like to see?[582.16] [582.16][S08] I would like to see somebody at a spaghetti eating[584.4] [584.4][S08]contest.[585.22] [585.22][S08]I would like their face to be dirty with sauce.[587.51] [587.51][S09] Excellent.[588.51] [588.51][S08] And I'd a whole hunk of spaghetti to be essentially[592.23] [592.23][S08]hanging from their mouth.[593.47] [593.47][S08]Sauce everywhere.[595.29] [595.29][S09] Sauce everywhere.[598.02] [598.02][S09]And camera pans in--[605.49] [605.49][S09]or camera pans out to see all of the contestants eating spaghetti[613.26] [613.26][S09]simultaneously.[614.292] [614.292][S08] Nice.[615.0] [615.0][S09] Yeah.[616.44] [616.44][S09]And we'll see how well the model can do at this one.[620.37] [620.37][S09]You can also use--[621.88] [621.88][S09]as I'm sure you can see here, you[623.61] [623.61][S09]can ground the initial model frame with an image.[628.78] [628.78][S09]So if you wanted to use something like Imagen 4[631.2] [631.2][S09]to generate an image as the seed frame for the video itself,[635.92] [635.92][S09]you would be able to do that as well.[638.73] [638.73][S08] So if you had, for example,[640.46] [640.46][S08]a picture of a spaghetti eating contest[642.61] [642.61][S08]and you wanted it to replicate where[644.23] [644.23][S08]the tables were, where the scenery was,[646.09] [646.09][S08]you could do that with this?[647.257] [647.257][S09] Absolutely.[648.298] [648.298][S09]And then there are some other capabilities and characteristics[650.95] [650.95][S09]of our latest Gemini models that allow[652.81] [652.81][S09]you to do really nice editing of images as well.[657.1] [657.1][S09]So perhaps you have a spaghetti eating contest,[659.87] [659.87][S09]but you only want to have a certain number of people[662.56] [662.56][S09]in the frame or you want to have a specific style of bench[667.06] [667.06][S09]or a specific kind of lighting.[668.66] [668.66][S09]You would also be able to give that image input to Gemini,[672.5] [672.5][S09]ask for it to transform it, and then use that as the seed[675.01] [675.01][S09]frame for the Veo 3.[676.12] [676.12][S08] So let me understand then.[677.703] [677.703][S08]What is new in Veo 3 that we didn't have before?[680.57] [680.57][S08]What makes it better?[681.92] [681.92][S09] Yeah.[682.712] [682.712][S09]So Veo 3 is better for a few reasons.[684.65] [684.65][S09]One is that it has the ability to produce[687.88] [687.88][S09]sounds, which has been really enchanting a lot of people,[690.68] [690.68][S09]I think.[691.51] [691.51][S09]Not just background noises, but also realistic music, dialogue,[697.75] [697.75][S09]all sorts of things that can really[699.49] [699.49][S09]construct videos that feel like they might[701.68] [701.68][S09]have been taken in real life.[704.56] [704.56][S09]Another way that we've improved Veo 3[707.14] [707.14][S09]is that the video outputs are grounded[709.45] [709.45][S09]in more physics understanding.[711.08] [711.08][S09]So as we look at videos, we can spot[714.22] [714.22][S09]ways in which the light or gravity really[717.19] [717.19][S09]does seem to align with the physical world.[719.83] [719.83][S09]And then there's also been a lot of improvements[722.32] [722.32][S09]around character consistency.[724.34] [724.34][S08] How are these additional features possible?[728.18] [728.18][S09] Yeah, I think DeepMind is really,[730.4] [730.4][S09]really paying attention to how the data is curated[733.69] [733.69][S09]to use to train the models.[735.16] [735.16][S09]When you think about it, you can see the word tree.[739.6] [739.6][S09]You can see a picture of a tree.[741.95] [741.95][S09]You could have the 3D representation of trees or any[749.41] [749.41][S09]of the other things that you would expect.[751.19] [751.19][S08] A sound of wind blowing through leaves.[753.893] [753.893][S09] Exactly.[754.81] [754.81][S09]And it's like a video of somebody panning around.[758.07] [758.07][S09]All of these things are still associated with that one entity,[762.78] [762.78][S09]but it's all very different modalities[765.05] [765.05][S09]describing the same thing.[766.73] [766.73][S09]And so I think historically folks have been concentrating[772.16] [772.16][S09]on just one modality-- so text, or code, or something similar--[776.03] [776.03][S09]when as humans, we experience the entire world[779.21] [779.21][S09]in very different ways.[781.14] [781.14][S09]Everything from seeing to hearing to touching,[784.14] [784.14][S09]all of these things.[785.52] [785.52][S09]And so I think that the team has put a lot of time,[789.11] [789.11][S09]and energy, and effort into being able to couple[791.63] [791.63][S09]together, not just the video footage, but also the sound that[796.22] [796.22][S09]composes the video footage, detailed descriptions, even[799.19] [799.19][S09]at the frame by frame level, and then also stitching all of that[804.02] [804.02][S09]together into a full representation of the training.[809.377] [809.377][S08] So whereas a language-only model might[811.46] [811.46][S08]have the word tree and its closely[813.84] [813.84][S08]associated with the word branch or twig.[816.13] [816.13][S08]The multimodal version not only has[818.58] [818.58][S08]all of those embedded within it, but additionally has[821.7] [821.7][S08]audio, images, video, all of these different layers.[825.648] [825.648][S09] Absolutely.[826.69] [826.69][S09]I think this is one of the things that[828.72] [828.72][S09]makes me most excited about the Gemini models is that we're[831.54] [831.54][S09]really the only model family that also allows you to output[835.41] [835.41][S09]text and code, but also images, to edit images,[838.65] [838.65][S09]to have output audio, which we'll see in a second,[841.83] [841.83][S09]as well as steerable audio.[843.67] [843.67][S09]So being able to say, speak softer, or speak more loud,[847.44] [847.44][S09]or speak in a different language.[849.81] [849.81][S09]All of these other model families[851.58] [851.58][S09]kind of relied on stitching together different trained[855.84] [855.84][S09]experiences, as opposed to baking it all into one[858.66] [858.66][S09]innate model.[860.31] [860.31][S09]And I think that's really powerful about Gemini.[864.868] [864.868][S08] Oh, OK.[865.66] [865.66][S08]It's ready.[866.28] [866.28][S09] Oh, my gosh.[866.98] [866.98][S09]Let's see it.[867.64] [867.64][S08] Go for it.[868.916] [868.916][CHATTER][870.66] [870.66][S08]I regret.[871.63] [871.63][S08]I regret.[872.37] [872.37][SLURPING IN VIDEO][873.25] [873.25][S09] Oh, my gosh.[876.364] [876.364][S09]So these do look actual people, though.[879.53] [879.53][S08] They do look actual people.[881.98] [881.98][S08]The spaghetti is-- I mean, I would say--[884.69] [884.69][SCREAMS IN VIDEO][885.97] [885.97][LAUGHTER][887.39] [887.623][S09] OK, we're going to pause for a second,[889.79] [889.79][S09]but look at the faces.[892.51] [892.51][S09]This does look a little bit more photorealistic than some[898.09] [898.09][S09]of the experiences that we've seen previously.[900.23] [900.23][S08] The hair especially.[901.563] [901.563][S08]If you just go to about there, two seconds in,[903.64] [903.64][S08]maybe, as it zooms out, this girl's hair.[906.98] [906.98][S08]I mean, look, she's even got a little bit of a--[909.46] [909.46][S08]you see the little parting bit, and tufts of baby hair.[912.478] [912.478][S09] Yeah.[913.27] [913.27][S08] But yeah, as she moves, that is really realistic.[916.13] [916.13][S09] This is so cool.[917.93] [917.93][S09]And then also the noises are something that I would imagine.[921.73] [921.73][S09]Yes, obviously disgusting, but things[923.59] [923.59][S09]that I would expect to see at a spaghetti eating contest.[926.77] [926.77][S09]And also how the sauce looks, how the noodles look,[933.75] [933.75][S09]that as you're gulping, some of the noodles[936.67] [936.67][S09]are falling back to the plate.[939.115] [939.115][S09]This is incredible.[940.752] [940.752][S08] And this is the notoriously difficult challenge[943.21] [943.21][S08]that people have used to see how good Veo is.[944.81] [944.81][S09] So instead of having one person eating[946.63] [946.63][S09]spaghetti, we just had 10 people eating spaghetti[948.88] [948.88][S09]all at the same table.[950.32] [950.32][S08] I mean, there was a lot of Will Smith eating[951.76] [951.76][S08]spaghetti for a number of years, I seem to remember.[954.08] [954.08][S09] Yes.[954.83] [954.83][S09]And so Will Smith eating spaghetti, that is so last year.[960.59] [960.59][S09]Let's try multiple people eating spaghetti.[962.51] [962.51][S08] The thing that I really[963.968] [963.968][S08]find impressive is how an individual object tracks[967.81] [967.81][S08]through the frames, and keeps its consistency.[971.41] [971.41][S08]Is that something that you have to impart afterwards?[974.6] [974.6][S09] I do think that the way that the model is[977.29] [977.29][S09]constructed, which is a diffusion model,[980.17] [980.17][S09]is attempting to hold that consistency throughout frames.[984.71] [984.71][S09]But a lot of this logic is baked into the model itself.[987.59] [987.59][S09]So in the before times, you would have to explicitly program[991.85] [991.85][S09]each one of the behaviors.[994.01] [994.01][S09]For now, I think that a lot of the magic behind the scenes[997.55] [997.55][S09]is just Veo and Gemini kind of figuring out[1000.94] [1000.94][S09]what they need to do in order to make the experiences[1004.15] [1004.15][S09]and make the outputs look photorealistic.[1006.07] [1006.07][S08] Because it has read the internet,[1008.24] [1008.24][S08]but it's also watched all of YouTube.[1009.86] [1009.86][S09] Well, not all of YouTube, and in addition[1012.22] [1012.22][S09]to a lot of synthetic data.[1013.55] [1013.55][S09]So as an example, we're incredibly[1018.01] [1018.01][S09]benefited by having so many game designers[1020.5] [1020.5][S09]and people from the game industry working at DeepMind.[1023.93] [1023.93][S09]And Demis, of course, has a game background.[1028.22] [1028.22][S09]But these environments are really[1030.97] [1030.97][S09]great synthetic data generators for all sorts of things.[1034.7] [1034.7][S09]You can have figures running through games,[1038.54] [1038.54][S09]accomplishing tasks, like, doing all sorts of behaviors[1042.412] [1042.412][S09]and monitoring them along the way, which[1044.079] [1044.079][S09]can be used as an excellent source of training data.[1046.589] [1046.589][S09]It's also nice in the sense that as you do gameplay,[1049.02] [1049.02][S09]you can have the camera orient itself in various ways[1052.46] [1052.46][S09]around to the characters.[1053.88] [1053.88][S08] So, OK, I know a lot of the buzz around this,[1056.255] [1056.255][S08]a lot of the new thing about Veo 3 is the audio.[1058.88] [1058.88][S08]Just, I mean, how is it being generated in order[1062.03] [1062.03][S08]to correspond?[1063.06] [1063.06][S08]Is it that it's generating something[1064.67] [1064.67][S08]that matches the visuals, or is it[1067.16] [1067.16][S08]like, there's a context which produces[1070.07] [1070.07][S08]both the visuals and the audio?[1071.667] [1071.667][S09] I think it's because the training data has[1074.0] [1074.0][S09]all of the different modalities associated with the thing[1078.47] [1078.47][S09]that it sees.[1079.14] [1079.14][S09]So it's not just seeing a video.[1082.07] [1082.07][S09]It also has the transcript.[1083.55] [1083.55][S09]It also has the frame by frame level description of the video[1086.24] [1086.24][S09]and what's happening.[1087.18] [1087.18][S09]It's also has the description of the audio,[1089.6] [1089.6][S09]if there's any background tracks.[1091.29] [1091.29][S09]And so all of that kind of brought together simultaneously[1095.63] [1095.63][S09]is capable of generating these much more immersive and natural[1099.74] [1099.74][S09]sounds and natural responses.[1102.77] [1102.77][S09]Because there are certainly instances[1104.87] [1104.87][S09]where if you listen to a song, you could read the sheet music[1112.79] [1112.79][S09]and you could hear the different tones[1116.12] [1116.12][S09]displayed, but it could also make you feel a certain way.[1119.28] [1119.28][S09]And so if I describe that I want to hear a song that reminds me[1126.29] [1126.29][S09]of being in a rainy café in Japan at nighttime,[1130.396] [1130.396][S09]that's something that you would really[1132.56] [1132.56][S09]want to have all of these different modalities coming[1135.65] [1135.65][S09]together to understand, as opposed to just guessing[1141.62] [1141.62][S09]at what a rainy café might sound like.[1143.502] [1143.502][S08] It's like a two-dimensional version[1145.46] [1145.46][S08]of a fully immersive experience.[1147.27] [1147.27][S09] Exactly.[1148.43] [1148.43][S08] That's nice.[1149.94] [1149.94][S09] I love that description of it as well.[1152.435] [1152.435][S09]We're getting closer and closer towards something that[1159.41] [1159.41][S09]feels very close to reality.[1161.158] [1161.158][S08] Like a simulated reality.[1162.7] [1162.7][S09] Yes.[1163.504] [1163.504][S09]And I don't think that was possible previously.[1166.93] [1166.93][S08] So thus far, then, is the Gemini app[1170.22] [1170.22][S08]that we know and love.[1171.16] [1171.16][S09] Yes.[1171.45] [1171.45][S08] But if you are a professional filmmaker,[1173.893] [1173.893][S08]or you want to take this a bit more seriously,[1175.81] [1175.81][S08]there is another place you can go to, correct?[1178.468] [1178.468][S09] Yes, it is called Flow.[1180.01] [1180.01][S09]It's built by our colleagues over in the Google Labs team,[1182.61] [1182.61][S09]and they've been partnering directly[1184.11] [1184.11][S09]with filmmakers to really build an experience that[1187.41] [1187.41][S09]aligns with their expectations.[1188.902] [1188.902][S08] So go on, show me Flow.[1190.36] [1190.36][S09] Yeah.[1192.39] [1192.39][S09]And so this is Flow, which is the first place that you would[1197.49] [1197.49][S09]go if you were a filmmaker.[1199.06] [1199.06][S09]You can also see some of the projects[1201.45] [1201.45][S09]that have been happening recently,[1204.48] [1204.48][S09]as well as this thing called Flow TV, which[1207.72] [1207.72][S09]is a really, really cool way to experience[1211.2] [1211.2][S09]some of the videos that have been generated by Flow recently.[1216.168] [1216.168][S08] So is the idea here then,[1217.71] [1217.71][S08]you still have the eight second videos,[1219.34] [1219.34][S08]but you can stitch them together?[1221.308] [1221.308][S09] You can stitch them together,[1223.1] [1223.1][S09]you can style them.[1224.51] [1224.51][S09]There are even camera controls associated.[1226.607] [1226.607][S08] Oh, wow.[1227.44] [1227.44][S09] So it really does give you a lot more creative[1230.23] [1230.23][S09]control as a filmmaker without necessarily having to--[1237.21] [1240.31][S09]oh, that's so cool, too.[1241.88] [1241.88][S08] These are--[1243.108] [1243.108][S09] But look at that.[1244.4] [1244.4][S09]The physics understanding of that video.[1246.7] [1246.7][S08] The fluid dynamics, right?[1249.04] [1249.04][S08]I mean, wow, extraordinary.[1252.608] [1252.608][S08]Look at that.[1253.15] [1253.15][S08]Oh, lord.[1253.65] [1253.65][S09] Yeah.[1254.442] [1254.442][S09]And then if you take a look at the prompt,[1256.22] [1256.22][S09]you can also see some of the sentences associated.[1260.12] [1260.12][S09]So "clean metal plastic injection opens releasing[1262.96] [1262.96][S09]a pinkish yellow jellyfish."[1266.26] [1266.26][S09]And a lot of the others along the way.[1269.11] [1269.11][S09]Yeah.[1269.71] [1269.71][S09]So detailed.[1270.89] [1270.89][S08] So detailed.[1272.12] [1272.12][S09] Yeah.[1272.29] [1272.29][S08] Show me the ways in which this differs[1274.373] [1274.373][S08]from the Gemini app, then.[1275.84] [1275.84][S09] Absolutely.[1276.882] [1276.882][S09]So this is a hyper specialized experience.[1280.27] [1280.27][S09]So if you click Create with Flow,[1282.24] [1282.24][S09]you're put into this development environment that you can use.[1286.48] [1286.48][S09]You can start a new project, where you can use Text to Video,[1291.3] [1291.3][S09]as well as Frames to Video, or Ingredients to Video.[1293.897] [1293.897][S08] So hold on, Text to Video,[1295.48] [1295.48][S08]that's the thing that we were already doing?[1296.65] [1296.65][S09] That's the thing that we were just doing.[1298.942] [1298.942][S09]There's also the frames to video.[1301.62] [1301.62][S09]So let's go ahead and do frames to video.[1305.94] [1305.94][S09]And you can stitch together multiple frames[1308.34] [1308.34][S09]to create a final scene.[1310.0] [1310.0][S09]You can also click this Camera Control button[1312.9] [1312.9][S09]and see these different examples to dollying in, dollying out,[1318.63] [1318.63][S09]static, or tilt down.[1320.97] [1320.97][S09]And all of those are things that can[1324.42] [1324.42][S09]build the takeaways that you would like[1327.81] [1327.81][S09]to see in the video footage.[1328.977] [1328.977][S08] So essentially this is just--[1330.685] [1330.685][S08]I mean, it's kind of the same thing, but just a more[1332.88] [1332.88][S08]specialized environment?[1334.0] [1334.0][S09] Definitely.[1335.042] [1335.042][S09]And we find that, just as you would[1336.68] [1336.68][S09]have specialized environments for musicians to create[1340.73] [1340.73][S09]their electronic tracks or CAD designers,[1344.72] [1344.72][S09]you would probably want a really,[1346.47] [1346.47][S09]really dedicated and focused UI for each one of these use cases[1350.0] [1350.0][S09]that can really hyper optimize for the things[1351.92] [1351.92][S09]that you would care about as a filmmaker.[1353.7] [1353.7][S08] I think the point about this[1355.367] [1355.367][S08]is that you then have this absolute open door[1357.86] [1357.86][S08]for creativity.[1358.83] [1358.83][S09] Yes.[1359.0] [1359.0][S08] So I've seen people take videos[1360.86] [1360.86][S08]as though the Spartans were Instagram influencers.[1363.98] [1363.98][S09] Yes.[1364.73] [1364.73][S09]Absolutely.[1365.23] [1365.23][S08] Like, reporting on their siege.[1368.758] [1368.758][S09] Absolutely.[1369.8] [1369.8][S09]And also character consistency across different experiences,[1374.91] [1374.91][S09]no matter what the lighting might be.[1376.58] [1376.58][S09]Like you might have a little monster character[1379.58] [1379.58][S09]that you want to have swimming through the ocean.[1381.918] [1381.918][S09]And then you also want to have him climbing a mountain,[1384.21] [1384.21][S09]and you want to have him singing on a stage.[1386.34] [1386.34][S09]And it's able to keep that same character consistent,[1389.04] [1389.04][S09]but to change all of the dynamics[1390.65] [1390.65][S09]around it, which is pretty magical.[1393.39] [1393.39][S08] I do wonder, though, about putting these tools[1396.03] [1396.03][S08]in the hands of people.[1397.26] [1397.26][S08]There are also concerns about it, too, right?[1399.73] [1399.73][S08]I mean, deepfakes, but also scams,[1404.0] [1404.0][S08]tricking people into thinking that news events are happening[1406.5] [1406.5][S08]that perhaps aren't.[1408.12] [1408.12][S08]Where do you stand on that?[1409.32] [1409.32][S09] Yes.[1410.07] [1410.07][S09]So we do have safety filters introduced within the Veo[1413.73] [1413.73][S09]models themselves.[1415.03] [1415.03][S09]And relatedly, for all of the Veo models[1417.66] [1417.66][S09]that are generated through the Gemini app,[1419.41] [1419.41][S09]there's a specialized watermark that gives you the ability[1423.0] [1423.0][S09]to know that this was AI authored, as opposed[1426.81] [1426.81][S09]to being something that was just shot via raw footage[1430.35] [1430.35][S09]out in the world.[1431.91] [1431.91][S09]But we also have special constraints in place[1434.82] [1434.82][S09]around not being able to generate images of things[1440.25] [1440.25][S09]like children or special entities.[1443.92] [1443.92][S09]There's also a constraint in place[1446.1] [1446.1][S09]such that government officials, or people[1448.77] [1448.77][S09]who are significantly present in the public sphere for policy,[1453.22] [1453.22][S09]or for science, or for any of the notable figures[1457.51] [1457.51][S09]in the world, you can't generate video content about them.[1462.4] [1462.4][S09]And even the kind of models that we experiment with internally,[1470.92] [1470.92][S09]they still have these constraints.[1472.61] [1472.61][S08] Well, if one of the key things about Veo 3[1474.95] [1474.95][S08]is the audio, can we go into the audio a bit more?[1477.19] [1477.19][S09] Yes.[1477.94] [1477.94][S09]We even just released a Gemini Text[1480.13] [1480.13][S09]to Speech API that allows you to generate audio,[1483.14] [1483.14][S09]including steerable audio, in multiple languages.[1485.205] [1485.205][S08] So without the images[1486.58] [1486.58][S08]to go with it, just audio only?[1488.27] [1488.27][S09] Just audio only, but really,[1490.1] [1490.1][S09]really expressive audio.[1491.78] [1491.78][S09]And you can also have multiple speakers in different languages.[1496.51] [1496.51][S09]So I believe you might have seen the NotebookLM[1500.05] [1500.05][S09]before with the podcasts that were generated.[1503.6] [1503.6][S09]This allows you to create customizable and similar[1508.24] [1508.24][S09]experiences using multiple speakers or a single speaker.[1511.173] [1511.173][S08] Well, let me just bring you back for a second,[1513.59] [1513.59][S08]because we did actually get to talk to the researchers who[1517.03] [1517.03][S08]were working on WaveNet.[1518.21] [1518.21][S09] Oh, yes.[1519.127] [1519.127][S08] This is like, now, I mean, only four years ago.[1522.12] [1522.12][S08]And this is where they were at that point, because what they[1524.62] [1524.62][S08]did is they trained a model, this was neural networks, right,[1528.85] [1528.85][S08]using my voice.[1530.19] [1530.19][S08]And this is where they got to.[1531.44] [1531.44][S01] Hi, there.[1532.01] [1532.01][S01]I'm a mathematician, author, and podcaster[1534.7] [1534.7][S01]who's fascinated by artificial intelligence.[1537.003] [1537.003][S08] It's very breathy, isn't it?[1538.67] [1538.67][S09] It is very breathy.[1539.72] [1539.72][S08] How have things changed since then?[1542.0] [1542.0][S09] Yes.[1542.75] [1542.75][S09]So things have changed significantly.[1544.54] [1544.54][S09]When WaveNet, which was pioneering at the time,[1546.86] [1546.86][S09]was first created, you would need[1549.49] [1549.49][S09]dedicated single-task models for each one of the things[1552.67] [1552.67][S09]that you were trying to do.[1554.0] [1554.0][S09]And so I started doing machine learning,[1556.4] [1556.4][S09]I think, around 2009, 2010, and it was extraordinarily painful[1560.71] [1560.71][S09]because you had to acquire all of these special purpose data[1564.13] [1564.13][S09]sets.[1564.8] [1564.8][S09]You had to get them cleaned up.[1566.72] [1566.72][S09]You would have to monitor for things like data drift.[1569.31] [1569.31][S09]If anything changed over time, you[1570.79] [1570.79][S09]would have to retrain the model from scratch.[1572.67] [1572.67][S09]And so WaveNet was a dedicated single-task model[1576.65] [1576.65][S09]for generating these really realistic, at this time,[1580.58] [1580.58][S09]sounding voices.[1582.11] [1582.11][S09]But it couldn't do other things.[1584.04] [1584.04][S09]So it couldn't have steerable audio.[1588.03] [1588.03][S09]Like, you couldn't say, "please, give me this audio clip in this[1595.04] [1595.04][S09]kind of style, and do it in German."[1598.11] [1598.11][S09]Whereas with our recent models, they[1600.89] [1600.89][S09]are a lot more steerable by design.[1603.96] [1603.96][S09]So you can give instructions about style,[1606.36] [1606.36][S09]about the language that you're speaking,[1610.22] [1610.22][S09]about pause instructions, or speak quickly, speak slower,[1615.98] [1615.98][S09]all sorts of things.[1617.25] [1617.25][S08] So how much of the WaveNet, I mean, code, even,[1620.72] [1620.72][S08]has actually ended up feeding into this model?[1623.49] [1623.49][S08]Or is it you started again once large language models[1627.202] [1627.202][S08]and transformers came on the scene?[1628.66] [1628.66][S09] Yeah.[1629.452] [1629.452][S09]So a lot of the code that was used to create the WaveNet[1632.52] [1632.52][S09]model, the architectures are a little bit[1636.81] [1636.81][S09]different for our Gemini family, but the data that was used[1641.34] [1641.34][S09]is definitely repurposed.[1642.94] [1642.94][S09]And so all of the examples created[1645.03] [1645.03][S09]for this is the text input, this is the audio output.[1648.39] [1648.39][S09]You can also enrich those kinds of data sets[1650.64] [1650.64][S09]with the descriptions of the style of the audio or the tone[1655.8] [1655.8][S09]or the temper of the voice.[1657.66] [1657.66][S09]All of that is incredibly useful for Gemini.[1660.22] [1660.22][S09]And that's been one of the really interesting things[1662.49] [1662.49][S09]to see over the years as well is that previously,[1666.31] [1666.31][S09]like with the first generation of Gemini models,[1668.8] [1668.8][S09]we had to do significant work to fine tune them.[1672.25] [1672.25][S09]So you had things like MetaLimb.[1675.1] [1675.1][S09]You had things like special purpose models for generating[1679.29] [1679.29][S09]code that were kind of variations of Gemini's.[1683.64] [1683.64][S09]And all of those training data have been now incorporated[1687.15] [1687.15][S09]into the base model itself.[1689.41] [1689.41][S09]So Gemini has just gotten better and better natively[1691.782] [1691.782][S09]at all of the tasks that people had previously needed[1693.99] [1693.99][S09]to fine tune to do.[1695.425] [1695.425][S08] So and then show me how it works.[1697.3] [1697.3][S08]Show me, give me some examples.[1698.71] [1698.71][S09] Exactly.[1699.627] [1699.627][S09]So if we go over to AI Studio, I'll open up a new tab[1703.59] [1703.59][S09]and pull us into the UI.[1705.64] [1705.64][S09]You can see here this great playground[1708.33] [1708.33][S09]for experimenting with and trying out the latest Gemini[1710.67] [1710.67][S09]models as soon as they're released.[1712.3] [1712.3][S08] And so to create audio,[1714.028] [1714.028][S08]it's the one that looks like an audio wave on the side.[1716.32] [1716.32][S09] So interestingly, for the text-to-speech model,[1719.32] [1719.32][S09]you would go to generate media, and then you[1721.5] [1721.5][S09]would go to Gemini speech generation,[1723.79] [1723.79][S09]and you're launched into this text-to-speech UI where[1726.57] [1726.57][S09]you can specify the different speakers, the different voices,[1730.12] [1730.12][S09]and then also the style instructions[1732.18] [1732.18][S09]for each one of the speakers.[1733.6] [1733.6][S08] Let's think of a prompt[1734.31] [1734.31][S08]then, because I'm particularly interested[1736.018] [1736.018][S08]in this different emotions.[1737.62] [1737.62][S08]So what about if you get it to say something like,[1739.95] [1739.95][S08]"I was waiting for you," and then we try different emotion.[1742.845] [1742.845][S09] All right, so we're[1744.22] [1744.22][S09]going to specify in the system instructions,[1747.04] [1747.04][S09]speak in a friendly tone like you're greeting[1754.27] [1754.27][S09]a relative who just came home.[1757.672] [1757.672][S08] Nice.[1758.38] [1758.38][S09] Yeah, and then start typing a prompt.[1760.87] [1760.87][S09]"I was waiting for you."[1763.09] [1763.09][S09]And what we're going to do is speak in a friendly tone,[1768.59] [1768.59][S09]only say what the person prompts you to say.[1778.03] [1778.03][S09]Yeah, and then hit Run.[1782.26] [1782.26][S05] I was waiting for you.[1784.48] [1784.48][S09] Amazing.[1784.84] [1784.84][S08] Friendly.[1785.36] [1785.36][S09] Yeah.[1786.01] [1786.01][S08] Can we try something different, though?[1787.18] [1787.18][S08]Can you make it more romantic?[1789.29] [1789.29][S09] Yes.[1790.21] [1790.21][S09]So speak in a romantic, hushed tone, very breathy.[1796.9] [1796.9][S09]And only say the words that the person puts into the prompt.[1805.94] [1805.94][S09]And so we'll see how this goes.[1808.5] [1808.5][S09]And it was, "I was waiting for you."[1810.66] [1810.66][S08] Yeah.[1811.52] [1811.52][S09] And hit Run.[1812.93] [1812.93][S09]And OK, good.[1815.13] [1815.13][S09]Zephyr is selected.[1817.94] [1817.94][S06] I was waiting for you.[1819.95] [1819.95][S09] Ooh.[1820.7] [1820.7][S08] Saucy.[1821.7] [1821.7][S09] Very saucy.[1822.8] [1822.8][S09]I also love that you can see in this UI the thoughts associated.[1827.39] [1827.39][S09]So Gemini's thinking process as it's[1830.48] [1830.48][S09]kind of going through the path of creating this audio response.[1833.46] [1833.46][S08] What has it said?[1834.27] [1834.27][S09] It says, "I've processed the input.[1836.312] [1836.312][S09]I've pinpointed the exact phrase I need to use.[1838.29] [1838.29][S09]The task core now centers on delivering the specific phrase.[1841.47] [1841.47][S09]I've meticulously identified the romantic, hushed,[1843.72] [1843.72][S09]and very breathy tone required, and the next step is generating[1846.56] [1846.56][S09]the audio."[1847.17] [1847.17][S09]So it really gives you step by step instructions[1850.82] [1850.82][S09]of how to incorporate all of these responses.[1852.695] [1852.695][S08] Can I do some more?[1853.987] [1853.987][S08]Can we do angry?[1854.66] [1854.66][S09] Yes.[1856.11] [1856.11][S09]You can definitely do angry.[1857.4] [1857.4][S09]So let's change the system instructions again.[1860.58] [1860.58][S09]And so let's say--[1862.823] [1862.823][S08] Someone's late for the date.[1864.49] [1864.49][S09] Yes.[1865.24] [1865.24][S09]So someone is late for the date.[1867.09] [1867.09][S09]I'm going to copy the system instructions again, turn off[1870.27] [1870.27][S09]the stream and start a new one.[1871.81] [1871.81][S09]And we will do this.[1874.96] [1874.96][S09]So speak in an annoyed, angry tone, like a person has just[1885.54] [1885.54][S09]arrived late for a date, and only say the words[1892.92] [1892.92][S09]that you see here.[1894.94] [1894.94][S09]And so let's try.[1895.648] [1895.648][S08] This is so much fun.[1896.982] [1896.982][S09] Yes, it is.[1898.08] [1898.08][S04] I was waiting for you.[1899.86] [1899.86][S09] Oh, that--[1901.0] [1901.0][S08] She's angry.[1901.78] [1901.78][S09] She is very angry.[1903.113] [1906.03][S08] Can you do grieving?[1907.632] [1907.632][S09] Grieving.[1908.59] [1908.59][S09]Yes.[1908.67] [1908.67][S08] Like a lost loved one?[1909.82] [1909.82][S09] Yes.[1910.57] [1910.57][S09]So let's try another stream and speak in a grieving tone.[1919.11] [1919.11][S09]Grieving tone, like you have just lost a loved relative.[1927.84] [1927.84][S09]Only say the words that the person puts into the prompt.[1930.67] [1930.67][S09]And "I was waiting for you."[1934.5] [1934.5][S09]And hit Run.[1935.09] [1938.97][S07] I was waiting for you.[1940.992] [1940.992][S09] Aw.[1941.7] [1941.7][S09]And so I feel like that was a little bit less grieving.[1946.57] [1946.57][S09]So you could probably experiment a little with the prompt.[1951.54] [1951.54][S08] Now, can you make them French?[1953.61] [1953.61][S09] So we can modify the system instructions.[1956.26] [1956.26][S09]So within this streaming real-time feature,[1958.98] [1958.98][S09]we can say something to the effect of only respond.[1962.46] [1962.46][S09]So only respond to the user in French.[1968.55] [1968.55][S09]Make sure to be bubbly and excited.[1972.672] [1972.672][S08] Nice.[1973.38] [1973.38][S09] Bubbly and excited.[1975.79] [1975.79][S09]And only say the text that the user adds in the prompt.[1982.72] [1982.72][S09]And so we can see what the output might be.[1987.375] [1987.375][S09]And we're still going to go with "I was waiting for you?"[1989.75] [1989.75][S08] Absolutely.[1990.31] [1990.31][S09] All right.[1991.31] [1991.31][S09]"I was waiting for you."[1993.85] [1993.85][S09]And we've got Zephyr selected.[1997.63] [1997.63][S09]We've got the Run button here, and let's try.[2001.52] [2004.36][S03] [SPEAKING FRENCH][2006.947] [2006.947][S09] I don't speak French.[2008.405] [2008.405][S08] [SPEAKING FRENCH][2010.29] [2010.29][S08]Yeah, I mean, sure.[2011.6] [2011.6][S09] Yeah, excellent.[2012.85] [2012.85][S08] So this is all in AI Studio?[2014.735] [2014.735][S09] This is all in AI Studio.[2016.36] [2016.36][S08] Which is available for me to play with?[2017.98] [2017.98][S09] Yes, it is available today for free[2020.022] [2020.022][S09]for people to experiment, to try things out.[2022.6] [2022.6][S09]And even cooler, if you're a developer,[2025.11] [2025.11][S09]there's a little button here called[2026.67] [2026.67][S09]Get SDK Code that whenever you click it,[2029.14] [2029.14][S09]it gives you everything that you just did in the UI,[2032.29] [2032.29][S09]but in code form.[2033.83] [2033.83][S09]So if you wanted it in Python, in TypeScript,[2037.36] [2037.36][S09]in any of the languages that you're[2039.04] [2039.04][S09]using in your personal project, all you[2040.96] [2040.96][S09]have to do to replicate what you just[2042.52] [2042.52][S09]did in AI Studio is click Get SDK Code,[2045.5] [2045.5][S09]and you're kind of off to the races.[2047.12] [2047.12][S08] Absolutely extraordinary.[2048.34] [2048.34][S08]Are there other things that you can do in AI Studio[2050.465] [2050.465][S08]that you can't do in the Gemini app?[2053.3] [2053.3][S09] So Gemini Live is kind[2055.69] [2055.69][S09]of Project Astra baked directly within the UI of AI Studio.[2059.527] [2059.527][S08] And Project Astra, of course, we[2061.36] [2061.36][S08]did a whole episode on this with Greg Wayne.[2063.949] [2063.949][S08]This is the real-time visual understanding, the first attempt[2068.8] [2068.8][S08]at a universal AI assistant.[2070.608] [2070.608][S09] Absolutely.[2071.65] [2071.65][S09]It can see anything that you see.[2073.297] [2073.297][S09]It can talk to you in real time.[2074.63] [2074.63][S09]It can talk to you in multiple languages.[2076.989] [2076.989][S09]And you can also add additional tool calls to this process.[2082.73] [2082.73][S09]So if you wanted Gemini to be able to pull out and extract[2085.63] [2085.63][S09]up-to-date information, if you wanted[2087.699] [2087.699][S09]it to be able to interact with any of the apps[2090.13] [2090.13][S09]or any of the kind of products that you[2092.739] [2092.739][S09]use every day, something like Calendar, or Docs, or Sheets,[2095.679] [2095.679][S09]or Gmail, you can give it access to be[2098.29] [2098.29][S09]able to use all of those features[2102.07] [2102.07][S09]within the context of the Gemini Live API.[2105.26] [2105.26][S08] OK, show me then, give me an example[2107.26] [2107.26][S08]then of this working.[2108.135] [2108.135][S09] Yeah.[2108.927] [2108.927][S09]So let's turn on Grounding with Google Search,[2110.95] [2110.95][S09]and then also Share Screen with Gemini.[2114.32] [2114.32][S09]So you can share your screen, you can share your Webcam.[2117.4] [2117.4][S09]And of course, you can Talk with the model.[2119.38] [2119.38][S09]But let's go ahead and share our screen.[2121.88] [2121.88][S09]And I'm going to select one of these Chrome tabs[2124.57] [2124.57][S09]that we currently have open.[2127.03] [2127.03][S09]And let's take a look at that video that we saw with Doug Eck[2130.84] [2130.84][S09]and see what it has to say.[2134.11] [2134.11][S09]Hey there, Gemini.[2135.14] [2135.14][S09]Could you tell me what you see on the screen?[2137.355] [2141.37][S03] I see a YouTube video playing.[2143.59] [2143.59][S03]A woman with red hair is speaking and subtitles are[2146.38] [2146.38][S03]displayed on the screen saying, "Hannah Fry,[2149.12] [2149.12][S03]welcome to 'Google DeepMind, The Podcast,' with me."[2151.98] [2151.98][S03]The video appears to be about AI and creativity.[2155.78] [2155.78][S09] Amazing.[2156.71] [2156.71][S09]So tell me a little bit about what you[2159.53] [2159.53][S09]think the video might cover.[2161.25] [2161.25][S09]And also what is Google DeepMind's latest[2164.9] [2164.9][S09]models for creative teams?[2166.48] [2170.9][S09]And so you can see here it's expanding out its thoughts,[2174.47] [2174.47][S09]and it's realizing that it needs to make a search.[2176.72] [2176.72][S03] --AI, your new creative muse,[2178.34] [2178.34][S03]and the speakers introduction of the "Google DeepMind,[2180.59] [2180.59][S03]The Podcast," it's likely the video explores the potential[2183.86] [2183.86][S03]of AI in creative fields.[2185.58] [2185.58][S03]It may feature discussions on how AI can inspire or assist[2188.78] [2188.78][S03]creative professionals.[2190.23] [2190.23][S03]Regarding Google DeepMind's latest models[2192.17] [2192.17][S03]for creative teams, they've introduced various tools,[2195.33] [2195.33][S03]including Veo 3 for video generation,[2197.72] [2197.72][S03]Imagen 4 for image creation, and Lyria 2 for music composition.[2201.96] [2201.96][S03]They've also developed Flow, an AI filmmaking tool,[2205.29] [2205.29][S03]and are working on world modeling AI models[2207.33] [2207.33][S03]with applications in gaming and entertainment.[2209.8] [2209.8][S03]These advancements aim to empower artists and creators.[2212.963] [2212.963][S09] Amazing.[2213.88] [2213.88][S08] Amazing.[2214.3] [2214.3][S09] Yeah.[2215.092] [2215.092][S09]And so one of the things that you can see here[2217.2] [2217.2][S09]is the thinking trace has been exposed.[2220.87] [2220.87][S09]So it's kind of walking through all of Gemini's thought process[2224.73] [2224.73][S09]on how to answer the question.[2226.39] [2226.39][S09]It realizes that it needs to invoke one of the tools[2229.23] [2229.23][S09]that it has access to, which is Google Search.[2231.73] [2231.73][S09]So it needs to look up some information.[2233.8] [2233.8][S09]Once it gets the responses back from its search,[2236.98] [2236.98][S09]it incorporates that back into its summary,[2239.14] [2239.14][S09]and then it generates this wonderful audio clip[2242.16] [2242.16][S09]with insights that it's captured around all of the products[2246.42] [2246.42][S09]that Google has recently released.[2248.11] [2248.11][S08] So this stuff, this Gemini Live stuff,[2249.79] [2249.79][S08]you also have it on Android phones, right?[2251.55] [2251.55][S08]So actually, I spotted this on my phone a little while ago[2254.85] [2254.85][S08]and have been using it extensively.[2258.15] [2258.15][S08]Let's give it a go.[2259.3] [2259.3][S08]Hang on.[2260.82] [2260.82][S08]What can you see right now, Gemini?[2265.12] [2265.12][S03] OK.[2265.82] [2265.82][S03]I see a woman with blonde hair sitting[2267.55] [2267.55][S03]at a desk with a microphone in front of her.[2269.81] [2269.81][S03]She has her arms crossed and is smiling.[2272.18] [2272.18][S03]There's a computer keyboard and mouse on the desk,[2274.48] [2274.48][S03]and a lamp in the background.[2275.96] [2275.96][S09] It's amazing.[2276.32] [2276.32][S03] What else can you tell me about--[2278.028] [2278.028][S08] That has been so useful in diagnosing[2280.27] [2280.27][S08]my sick houseplants, I can't tell you.[2281.993] [2281.993][S09] Oh, wow.[2282.91] [2282.91][S09]And it's also just been so cool to have something[2286.06] [2286.06][S09]that feels like a real life commentator[2290.74] [2290.74][S09]on as you go about your daily life.[2293.57] [2293.57][S09]There's been something really, really[2297.13] [2297.13][S09]fascinating too about using it as a pair programmer.[2300.82] [2300.82][S09]One thing that I commonly use this[2302.86] [2302.86][S09]for is we have a tool at Google called Google Colab,[2307.43] [2307.43][S09]which is a notebook-spaced environment with Python code.[2310.72] [2310.72][S09]You can run code, you can execute it.[2314.86] [2314.86][S09]But Gemini also understands Colab features.[2319.97] [2319.97][S09]So if we want to share this tab instead, you could even say,[2326.44] [2326.44][S09]"hey, Gemini, what do you see on the screen now?[2329.3] [2329.3][S09]And if you see any code, please explain what it does."[2332.51] [2332.51][S08] Oh, nice.[2333.5] [2333.5][S09] Yeah.[2334.292] [2337.48][S03] I see a Google Colab notebook on the screen titled,[2340.84] [2340.84][S03]"Making the Most of Your Colab Subscription."[2343.31] [2343.31][S03]It seems to provide tips for optimizing Colab usage,[2346.52] [2346.52][S03]focusing on faster GPUs and more memory.[2349.37] [2349.37][S03]There are a couple of code snippets visible.[2351.62] [2351.62][S03]The first one appears to check if a GPU is connected[2354.49] [2354.49][S03]and display GPU information.[2356.54] [2356.54][S03]The second code uses the psutil library[2359.5] [2359.5][S03]to determine the total amount of RAM available in the runtime,[2362.48] [2362.48][S03]and presents that in gigabytes.[2363.795] [2363.795][S09] So it's like having[2365.17] [2365.17][S09]a helpful assistant that somehow understands every single thing--[2368.317] [2368.317][S08] That you could possibly be looking at.[2370.4] [2370.4][S09] Absolutely.[2371.442] [2371.442][S09]And that also is capable of answering,[2375.85] [2375.85][S09]in an empathetic way, all of the questions that you might have[2380.038] [2380.038][S09]or that a curious human might have about the things that they[2382.58] [2382.58][S09]see every day.[2383.4] [2383.4][S08] So this is good.[2384.06] [2384.06][S08]I mean, I already use Google Colab with my students.[2386.19] [2386.19][S09] Oh, amazing.[2387.273] [2387.273][S08] So thank you for that.[2388.7] [2388.7][S08]That's going to be very helpful.[2390.033] [2390.033][S08]But for people who can't necessarily code themselves,[2394.22] [2394.22][S08]I mean, there's another offering within AI Studio.[2396.428] [2396.428][S09] Absolutely.[2397.47] [2397.47][S09]So in addition to being able to take the Gemini APIs[2401.48] [2401.48][S09]and embed them within tools like Cursor, or Windsurf, or Copilot,[2406.64] [2406.64][S09]or any of the other many coding IDEs[2409.432] [2409.432][S09]that you might have access to if you're attempting to write code,[2412.14] [2412.14][S09]we also have something called the new Build[2414.32] [2414.32][S09]feature that allows you to build apps directly with Gemini.[2418.19] [2418.19][S09]So the Build Apps with Gemini section[2421.7] [2421.7][S09]is unique in that all of the code that[2424.37] [2424.37][S09]gets generated within this UI is hyper[2426.56] [2426.56][S09]optimized for the latest SDKs, the latest code[2429.83] [2429.83][S09]that we have for interfacing with the Gemini models.[2432.677] [2432.677][S08] So just to put that another way,[2434.51] [2434.51][S08]even if you have never written a line of code in your life,[2438.33] [2438.33][S08]you are going to have slick code if you just prompt Gemini here[2441.827] [2441.827][S08]to build you an app.[2442.66] [2442.66][S09] Absolutely.[2443.13] [2443.13][S09]And it will be using the latest models, the latest[2445.38] [2445.38][S09]features from the models.[2447.12] [2447.12][S09]It will be generating really, really robust TypeScript code,[2451.33] [2451.33][S09]and it will also be resolving any errors along the way.[2454.99] [2454.99][S09]So if the model hits any problems, it's able to cycle[2457.89] [2457.89][S09]back through, fix the error to get the models[2460.62] [2460.62][S09]kind of implementation of an app and a clear and coherent state.[2463.94] [2463.94][S08] Self-healing code.[2465.19] [2465.19][S09] Self-healing code, and all with you just[2468.12] [2468.12][S09]having to describe what you would like to see.[2470.61] [2470.61][S09]This is interesting too in the sense[2472.5] [2472.5][S09]that a lot of the vibe coding demos[2475.59] [2475.59][S09]that you might have seen externally[2477.06] [2477.06][S09]are really, really focused on just creating a simple app.[2480.07] [2480.07][S09]So something that's not AI-enriched.[2482.31] [2482.31][S09]This is hyper optimized for creating[2484.65] [2484.65][S09]apps that are also using the Gemini[2486.42] [2486.42][S09]APIs, or Imagen, or the like.[2488.82] [2488.82][S09]So it's really, really cool to see[2490.77] [2490.77][S09]what you can cook up using the Build Apps with Gemini.[2494.29] [2494.29][S09]If we have an idea for an app around creating a--[2498.742] [2498.742][S08] Spaghetti competitions.[2500.2] [2500.2][S09] Spaghetti competitions.[2501.742] [2501.742][S09]This is also correct.[2502.65] [2502.65][S09]So maybe we say, like, generate an application that[2509.61] [2509.61][S09]showcases a spaghetti competition or spaghetti eating[2514.95] [2514.95][S09]contest.[2516.42] [2516.42][S09]Make sure to use Imagen to generate[2523.08] [2523.08][S09]example images for the website landing photo,[2528.97] [2528.97][S09]as well as other photos about the competition.[2534.6] [2534.6][S09]Also, make sure to include a sign-up form for people[2541.65] [2541.65][S09]to indicate interest.[2543.263] [2543.263][S08] So this is essentially[2544.68] [2544.68][S08]going to make a website where all of the images[2547.44] [2547.44][S08]are AI-generated.[2549.91] [2549.91][S09] All of the images are AI-generated.[2552.39] [2552.39][S09]It should have a form where people[2554.1] [2554.1][S09]can submit their information, if they would like to see it.[2557.59] [2557.59][S09]And it also has a kind of immediately launched us[2561.64] [2561.64][S09]into this new IDE with a code assistant, kind of a directory[2565.78] [2565.78][S09]structure here in the middle, and then[2567.64] [2567.64][S09]also a place for you to visualize the app as it gets[2571.12] [2571.12][S09]created here off to the right.[2572.63] [2572.63][S09]And so this thinking trace box, if we expand it out,[2575.72] [2575.72][S09]you can see that Gemini is doing a lot of work behind the scenes[2579.25] [2579.25][S09]to build the kind of core features of the app,[2583.03] [2583.03][S09]to map out the structure, to build the components.[2587.78] [2587.78][S09]And it's writing every single one of these code files[2591.37] [2591.37][S09]into being as we sit here watching, including everything[2596.92] [2596.92][S09]from design patterns, to things like the latest React libraries,[2604.09] [2604.09][S09]to even doing error handling for you,[2607.03] [2607.03][S09]so you don't have to think through all of the things[2609.28] [2609.28][S09]that you would have to consider if building out[2612.13] [2612.13][S09]a native application.[2613.295] [2613.295][S08] OK, so I am someone who has built[2615.17] [2615.17][S08]websites in the past.[2616.52] [2616.52][S08]And honestly, even with the tools[2619.478] [2619.478][S08]that are supposed to make it quicker,[2621.02] [2621.02][S08]weeks and weeks, weeks and weeks and weeks this would take.[2624.14] [2624.14][S08]And my websites were rubbish.[2625.495] [2625.495][LAUGHTER][2626.96] [2626.96][S08]Here we go.[2627.777] [2627.777][S09] Oh, my gosh.[2628.86] [2628.86][S08] It's chosen Comic Sans.[2630.42] [2630.42][S08]I love it.[2631.02] [2631.02][S09] It absolutely has chosen Comic Sans.[2633.103] [2633.103][S09]And so it has the contest.[2635.93] [2635.93][S09]It's got some images that it looks like are[2639.35] [2639.35][S09]in the process of loading.[2640.89] [2640.89][S08] Oh, look at this.[2642.18] [2642.18][S09] Oh, my gosh.[2643.41] [2643.41][S09]And then if I give it a little bit more room,[2645.853] [2645.853][S09]so let's hide this code assistant.[2647.27] [2647.27][S08] A fake trophy.[2647.94] [2647.94][S09] Yeah, it's got a fake trophy.[2649.98] [2649.98][S09]It's got the contest, including a background image[2652.43] [2652.43][S09]that it generated of the contest,[2653.9] [2653.9][S09]and then also changed the transparency of the image[2658.58] [2658.58][S09]so that it matched this very interesting stylistic choice[2662.3] [2662.3][S09]that it's created.[2664.35] [2664.35][S09]And it also has some contest highlights[2666.8] [2666.8][S09]for people who it looks have previously[2671.42] [2671.42][S09]won the contest before.[2673.14] [2673.14][S09]And all of these images, by the way, they're using Imagen 3.[2676.31] [2676.31][S09]So not Imagen 4.[2678.215] [2678.215][S08] But you could also, at this point[2680.09] [2680.09][S08]then, let's say that there was, I don't know,[2681.965] [2681.965][S08]some video of a previous spaghetti eating competition,[2685.01] [2685.01][S08]and you wanted to have a page, which[2686.84] [2686.84][S08]was a story about that moment.[2688.91] [2688.91][S08]You could upload a YouTube video,[2691.16] [2691.16][S08]have Gemini Live essentially watch it for you,[2695.03] [2695.03][S08]summarize it, and then embed it into the website?[2697.443] [2697.443][S09] Absolutely.[2698.485] [2698.485][S09]You could do all of that work.[2700.44] [2700.44][S09]You could even use the Gemini search capabilities.[2703.53] [2703.53][S09]So finding examples of related information[2707.93] [2707.93][S09]to spaghetti eating competitions and citing[2710.12] [2710.12][S09]all of those sources for the video itself.[2711.88] [2711.88][S08] And if you wanted to turn this[2713.63] [2713.63][S08]into an actual website, what do you do then?[2717.24] [2717.24][S09] So to turn it into an actual website, something[2719.93] [2719.93][S09]that you could share with your friends[2721.513] [2721.513][S09]and not just share the file itself, what you can do[2724.85] [2724.85][S09]is there's this little rocket here off[2726.83] [2726.83][S09]to the side that says Deploy to Cloud Run.[2730.53] [2730.53][S09]Since this has been integrated so closely with the Google Cloud[2734.64] [2734.64][S09]kind of development environment, what you can do[2737.97] [2737.97][S09]is say Deploy to Cloud Run.[2740.46] [2740.46][S09]And so after the app gets created,[2743.89] [2743.89][S09]you can select a project.[2745.66] [2745.66][S09]So just one of the projects that you have access[2748.26] [2748.26][S09]to via Cloud Run.[2752.1] [2752.1][S09]It verifies the project that you could actually use, and then[2755.7] [2755.7][S09]says that you can deploy it, and gives you[2758.04] [2758.04][S09]a unique URL that you can share with friends that's scalable.[2762.13] [2762.13][S09]And that, even more importantly, is hiding all of those APIs keys[2765.45] [2765.45][S09]so that you can use them and your app can use them,[2767.65] [2767.65][S09]but nobody else would be able to capture them.[2770.188] [2770.188][S08] What does this mean for developers, though?[2772.48] [2772.48][S08]I mean, you've just done some really extraordinary things[2775.29] [2775.29][S08]very, very quickly.[2776.49] [2776.49][S09] Yes.[2777.24] [2777.24][S09]Like, I do feel like for developers, this[2780.27] [2780.27][S09]means that they can focus more on building,[2782.22] [2782.22][S09]and ideation, and the product experience, as opposed to--[2786.33] [2786.33][S09]the daily life of a developer right[2789.28] [2789.28][S09]now is a lot of things that aren't necessarily[2792.82] [2792.82][S09]the most exciting in the world.[2794.69] [2794.69][S09]So you might be upgrading a code base[2796.48] [2796.48][S09]from one version to another, or you[2799.3] [2799.3][S09]might be adding typing to a repository,[2802.52] [2802.52][S09]or you might need to periodically check[2805.063] [2805.063][S09]for security vulnerabilities and make sure your code base aren't[2807.73] [2807.73][S09]susceptible to them.[2809.53] [2809.53][S09]And all of these things, they aren't joyful.[2812.93] [2812.93][S09]It's kind of tidying up an apartment[2815.17] [2815.17][S09]to make sure that your app is kind of sustainable[2818.53] [2818.53][S09]and maintained.[2820.0] [2820.0][S09]And there's that saying that everybody wants to build[2822.7] [2822.7][S09]and nobody wants to do maintenance.[2824.27] [2824.27][S09]I think one of the promises of these models[2826.87] [2826.87][S09]and these kinds of capabilities is[2828.97] [2828.97][S09]that there are so many more opportunities for software[2831.91] [2831.91][S09]developers to build and to really[2835.09] [2835.09][S09]build more ambitious systems.[2837.52] [2837.52][S09]And even more importantly, it opens up[2839.8] [2839.8][S09]the door for more people to learn and get[2842.65] [2842.65][S09]inspired by this whole process of creating[2845.9] [2845.9][S09]and getting something out into the world.[2847.88] [2847.88][S08] Do you think all of these tools[2849.02] [2849.02][S08]together then, do you think that they will fundamentally[2851.6] [2851.6][S08]change the way we think about human creativity?[2853.978] [2853.978][S09] Absolutely.[2855.02] [2855.02][S09]Like, human creativity is about to have[2858.14] [2858.14][S09]this explosion of progress.[2861.45] [2861.45][S09]And there's this promise of everyone[2863.36] [2863.36][S09]being able to become a creator, and not just a creator in one[2867.8] [2867.8][S09]specific discipline, but to also be[2869.57] [2869.57][S09]able to expand to that out into many other disciplines.[2872.82] [2872.82][S09]So I think that there's real magic in having people[2876.8] [2876.8][S09]who are scientists maybe in the physical sciences,[2879.56] [2879.56][S09]or chemistry or biology, or people who are historians[2883.67] [2883.67][S09]or musicians, being able to suddenly get[2887.03] [2887.03][S09]all of their ideas or their projects[2890.0] [2890.0][S09]into a digital form that could be shared with others.[2892.85] [2892.85][S08] Paige, absolutely fascinating.[2894.6] [2894.6][S08]Thank you so much for joining me.[2895.95] [2895.95][S09] It was awesome to be here.[2897.617] [2897.617][S09]Thank you so much.[2898.44] [2898.44][S08] With these new tools,[2900.06] [2900.06][S08]it feels like suddenly everything has clicked together.[2902.96] [2902.96][S08]Until now, we've spoken to researchers[2904.64] [2904.64][S08]about every individual element of these new releases, audio,[2908.82] [2908.82][S08]video, language.[2910.8] [2910.8][S08]But there is something so different about having[2913.52] [2913.52][S08]all of those elements integrated and working seamlessly together.[2917.19] [2917.19][S08]And I have been coming here for years.[2919.67] [2919.67][S08]I have spoken to the people all along the way.[2922.43] [2922.43][S08]But even still, seeing these tools come to life[2927.11] [2927.11][S08]and imagining the possibilities, I still feel completely wowed.[2930.69] [2930.69][S08]And to be honest with you, I'm now[2932.78] [2932.78][S08]itching to get on the train home and just[2934.488] [2934.488][S08]try out all of the things that I've always wanted to build,[2936.947] [2936.947][S08]but never had time to.[2937.95] [2937.95][S08]You have been listening to "Google DeepMind, The Podcast,"[2940.55] [2940.55][S08]with me, Professor Hannah Fry.[2942.2] [2942.2][S08]If you enjoyed this episode, then[2943.94] [2943.94][S08]do subscribe to our YouTube channel[2945.8] [2945.8][S08]or leave a review on your favorite podcast platform.[2948.45] [2948.45][S08]And of course, we have plenty more episodes[2951.32] [2951.32][S08]on a whole range of topics to come, so do--[2954.29] [2954.29][MUSIC PLAYING][2957.94]