[1.458][MUSIC PLAYING][4.86] [6.804][S01] Welcome to "Google DeepMind-- the Podcast."[9.7] [9.7][S01]I'm Professor Hannah Fry.[11.01] [11.01][S01]Now, I want to start today, unusually,[13.09] [13.09][S01]perhaps with a clip from another podcast.[16.17] [16.17][S01]Listen to this.[16.83] [16.83][AUDIO PLAYBACK][17.497] [17.497][S01]- What's the overall message here?[19.06] [19.06][S01]Is it social commentary, artistic expression, or just[23.34] [23.34][S01]a really elaborate joke?[24.55] [24.55][S01]- That's the beauty of this piece, I think.[26.46] [26.46][S01]It defies easy categorization.[28.87] [28.87][S01]It exists in this liminal space between language[32.009] [32.009][S01]and non-language, between art and absurdity.[34.007] [34.007][END PLAYBACK][34.59] [34.59][S01] This is a very interesting discussion,[36.673] [36.673][S01]which, as you might have guessed, is AI generated.[39.37] [39.37][S01]But what is notable about this particular clip,[42.055] [42.055][S01]aside from the fact that neither of the two podcast hosts[44.43] [44.43][S01]have ever existed, is that their conversation,[47.52] [47.52][S01]a mini treatise on human nature and our relationship with art,[51.3] [51.3][S01]was generated from the most unusual of prompts.[55.26] [55.26][S01]The podcast itself was created by a new feature[58.26] [58.26][S01]called Audio Overview, part of NotebookLM,[61.38] [61.38][S01]a personalized AI research assistant from Google Labs.[64.68] [64.68][S01]Now, NotebookLM is powered by Gemini,[67.04] [67.04][S01]and it lets you upload your sources, anything from PDFs[71.47] [71.47][S01]to videos to generate insights, explanations and, of course,[75.77] [75.77][S01]podcasts.[76.79] [76.79][S01]We often think of AI as just crunching through data[79.9] [79.9][S01]and spitting out answers, but NotebookLM[82.72] [82.72][S01]draws on expertise from storytelling[85.36] [85.36][S01]to present information in an engaging way.[88.46] [88.46][S01]And we wanted to see what happens[90.19] [90.19][S01]when you ask NotebookLM to analyze what most people would[94.63] [94.63][S01]consider to be nonsense, a single document containing[98.17] [98.17][S01]just two words repeated a thousand times over--[101.8] [101.8][S01]"cabbage" and "puddle."[104.12] [104.12][S01]And here is the result.[105.27] [105.27][AUDIO PLAYBACK][105.65] [105.65][S01]- So I have to admit, at first, I was like,[107.442] [107.442][S01]what is going on here?[110.87] [110.87][S01]But the more I think about it, [LAUGHS] the curious I get.[114.28] [114.28][S01]- It is fascinating, isn't it?[116.3] [116.3][S01]We're like, dealing with this one-piece puzzle.[118.37] [118.37][S01]- Right[118.91] [118.91][S01]- And we're trying to figure out, well,[119.93] [119.93][S01]what does this piece tell us?[121.54] [121.54][S01]What do you think?[122.39] [122.39][S01]What's your first impression?[123.89] [123.89][S01]- Honestly, it's almost like hypnotic or something.[128.03] [128.03][S01]If you were really staring into a puddle and all[130.36] [130.36][S01]you saw were these cabbages floating around--[132.355] [132.355][S01]- I can see it.[132.98] [132.98][S01]- --it's a little unsettling but also kind of funny.[135.222] [135.222][END PLAYBACK][135.805] [135.805][S01] Several minutes of intellectual analysis[138.04] [138.04][S01]packed to the brim with seemingly relevant ideas that[141.43] [141.43][S01]are nowhere in the original document.[144.61] [144.61][S01]It's actually quite impressive, really.[147.8] [147.8][S01]I am joined today by two people who are deeply involved[150.64] [150.64][S01]in writing NotebookLM story.[152.51] [152.51][S01]Joining us from San Francisco is Steven Johnson,[155.06] [155.06][S01]NotebookLM's editorial director and also a "New York Times"[158.8] [158.8][S01]best-selling author.[160.13] [160.13][S01]And in Mountain View, California,[161.81] [161.81][S01]Raiza Martin is a senior product manager for AI at Google Labs[165.46] [165.46][S01]who leads the team behind NotebookLM.[167.74] [167.74][S01]Welcome to the podcast, both of you.[169.84] [169.84][S01]Now, I want to start with the feature[171.76] [171.76][S01]that everybody's been talking about, this Audio Overview.[175.61] [175.61][S01]And well, I understand that you've got a little clip[178.36] [178.36][S01]that you want to play me.[180.02] [180.02][S03] Yes, let's play the clip.[181.92] [181.92][S03]I think you will enjoy this, Hannah.[183.12] [183.12][S01] OK, here we[183.47] [183.47][AUDIO PLAYBACK][184.137] [184.137][S01]- Welcome back, everyone.[185.34] [185.34][S01]Ready for another deep dive?[187.04] [187.04][S01]Today, we're shrinking down, way down.[190.71] [190.71][S01]- Microscopic, you might say?[192.11] [192.11][S01]- Exactly.[192.61] [192.61][S01]Think about those tiny little droplets of water,[195.732] [195.732][S01]you know, like the ones you see on a freshly washed car.[198.065] [198.065][S01]- Oh yeah.[198.84] [198.84][S01]- But imagine those droplets clinging to an airplane wing.[201.257] [201.257][S01] [LAUGHS] Oh my gosh.[202.59] [202.59][S01]- Or on a plant leaf.[203.6] [203.6][S01]- Right.[204.1] [204.1][S01]Being sprayed with pesticides.[206.39] [206.39][S01]The way those droplets behave--[207.92] [207.92][S01] Oh, you guys.[208.22] [208.22][S01]- --is actually incredibly important for all kinds[210.65] [210.65][S01]of things--[211.19] [211.19][S01]- It is?[211.76] [211.76][S01]- Making planes safer.[213.06] [213.06][S01]- More efficient farming.[214.5] [214.5][S01]- Even figuring out how rain forms.[216.92] [216.92][S01]- Wow, that's fascinating.[218.73] [218.73][S01]- We're diving into some serious research today.[220.73] [220.73][END PLAYBACK][221.313] [221.313][S01] That was my PhD, the first page of my PhD thesis.[224.67] [224.67][S01]Extraordinary.[225.86] [225.86][S01]I mean, frankly, there is no good stuff,[227.78] [227.78][S01]apart from heavy equations in math.[230.31] [230.31][S01]OK.[230.81] [230.81][S01]Lots of things to notice about that.[232.59] [232.59][S01]For starters, they made it sound much more[234.47] [234.47][S01]exciting than it actually is.[236.585] [236.585][S03] That's the point.[237.96] [237.96][LAUGHS][238.82] [238.82][S01] But also, though, the sort of back and forth.[241.51] [241.51][S01]I mean, the two voices, they were finishing[243.66] [243.66][S01]each other's sentences.[244.75] [244.75][S01]It felt very fluid, pardon the pun, very natural.[249.707] [249.707][S03] Imagine defending your dissertation now.[252.04] [252.04][S03]You could just play the podcast and leave it at that, I think,[255.93] [255.93][S03]if you'd only had that at your disposal back then.[258.22] [258.22][S01] Raiza, have you been surprised by people's reaction[260.1] [260.1][S01]to this?[260.6] [260.6][S01]Because it's had really quite serious uptake, hasn't it?[263.73] [263.73][S02] Yes, and I think the most surprising thing to me,[266.92] [266.92][S02]and, really, equally delightful, is how people are using it.[271.18] [271.18][S02]I think I imagined how they might.[273.28] [273.28][S02]But I think the beautiful thing about launching something[275.91] [275.91][S02]with this much sort of excitement around it[278.1] [278.1][S02]is you see a whole new universe of what everybody has been[281.85] [281.85][S02]trying, from things that are funny,[283.42] [283.42][S02]things that are entertaining, things that are inspiring[286.17] [286.17][S02]or really meaningful.[287.8] [287.8][S02]It's just been incredible.[289.32] [289.32][S02]I actually probably spend a good chunk[291.045] [291.045][S02]of my day, a third of my day, just listening to these.[293.295] [293.295][LAUGHS][293.64] [293.64][S03] [LAUGHS] Really?[294.3] [294.3][S01] You set up a Discord server, didn't you,[296.467] [296.467][S01]just to let people share stories about the ways[298.763] [298.763][S01]that they're using it.[299.68] [299.68][S01]What kind of things have come up?[301.81] [301.81][S03] So that was an interesting example,[303.992] [303.992][S03]playing your dissertation, because one of the things that I[306.45] [306.45][S03]think genuinely surprised us is people[308.43] [308.43][S03]would put their CVs and their resumes in there,[311.01] [311.01][S03]and it was almost like a little like hype machine.[314.2] [314.2][S03]If you were feeling down about yourself,[316.12] [316.12][S03]you would listen to like a 10-minute audio conversation[320.58] [320.58][S03]between two very enthusiastic hosts.[322.29] [322.29][S03]You're like, wow, Steven has really done a lot in his career.[325.91] [325.91][S03]It's very impressive.[327.14] [327.14][S03]But actually, a more serious version of that--[329.39] [329.39][S03]I mean, that's kind of fun and playful,[331.22] [331.22][S03]but people are using it, like, you can kind of workshop[335.02] [335.02][S03]things you're working on.[336.59] [336.59][S03]So you can upload a short story you're working on[339.01] [339.01][S03]and say, hey, give me some constructive criticism on this.[343.09] [343.09][S03]And you listen to people talking about your work[347.65] [347.65][S03]and they're very good at pulling out[350.32] [350.32][S03]the kind of interesting twists or focusing[352.48] [352.48][S03]on the characters that are particularly compelling or not.[356.29] [356.29][S03]And so it's a way of getting a little--[358.0] [358.0][S03]it's almost like a little focus group for stuff[360.37] [360.37][S03]that you're working on, which is really amazing.[362.95] [362.95][S01] I guess also hearing people actually[364.99] [364.99][S01]talk about it out loud adds that kind of extra layer of, I don't[368.56] [368.56][S01]know, objectivity almost, Raiza.[371.17] [371.17][S02] I would say it's been really surprising[376.75] [376.75][S02]because if we think about it, a lot of the content or content[380.11] [380.11][S02]generation, if you just render it in text, it's not new.[384.1] [384.1][S02]It's like if I upload my CV and then I[386.8] [386.8][S02]have an LLM spit out something that[388.72] [388.72][S02]says like, oh, here's Raiza's career,[390.62] [390.62][S02]write a summary of sorts--[391.97] [391.97][S02]maybe there's a few interesting tidbits[393.79] [393.79][S02]that it pulls out here and there--[395.57] [395.57][S02]that was novel two years ago, and everybody[398.17] [398.17][S02]was excited by that.[399.41] [399.41][S02]But I think adding that new layer or that new modality[403.09] [403.09][S02]of just very human-like voices, I[405.94] [405.94][S02]think it connects with people in a very different way.[411.28] [411.28][S02]Personally, I call this type of technology human-like, where[415.66] [415.66][S02]you recognize it as being very similar to you[418.72] [418.72][S02]and it resonates with you in a different way as a result.[421.76] [421.76][S02]And I think the first time I listened to my CV,[425.33] [425.33][S02]I knew what to expect.[426.68] [426.68][S02]But when I heard it, I still felt that bubble inside of me,[430.28] [430.28][S02]like the woo![431.47] [431.47][S02][LAUGHS] And I think that's the magic of new modalities.[435.37] [435.37][S03] I think the other point on this[437.5] [437.5][S03]is that human beings have been learning and exchanging[440.26] [440.26][S03]information through conversation for hundreds of thousands[443.77] [443.77][S03]of years.[444.53] [444.53][S03]We've been learning by reading structured text[447.04] [447.04][S03]on a page for 500 years, and structured text[451.06] [451.06][S03]on a screen for 30 years.[454.72] [454.72][S03]And so when you activate that sense of a genuine human-like[458.83] [458.83][S03]conversation, it's just a deep, ancient kind of ancestral part[464.05] [464.05][S03]of who we are that--[466.06] [466.06][S03]I think that's one of the reasons why it just[468.64] [468.64][S03]lights up people when they hear it for the first time.[470.96] [470.96][S01] Also interesting, I think[471.76] [471.76][S01]that you decided to have two hosts rather[473.71] [473.71][S01]than just one person sort of talking into space, as it were,[477.32] [477.32][S01]which--[477.82] [477.82][S01]I guess it speaks to the point that you're making, Steven.[480.58] [480.58][S03] Yeah, it's just a very different format.[483.2] [483.2][S03]If you just have one person, it feels like text to speech,[485.85] [485.85][S03]right?[486.35] [486.35][S03]We've heard text to speech before.[487.767] [487.767][S03]You're just like, the computer is turning the text that it just[490.45] [490.45][S03]wrote into something I can listen to, which is great.[494.12] [494.12][S03]And we're interested in trying to figure out ways we[496.57] [496.57][S03]can do that in other formats.[498.142] [498.142][S03]But to get the conversation right--[499.6] [499.6][S03]and we can dive into this in more detail--[501.47] [501.47][S03]there are all these subtle things[502.9] [502.9][S03]that you have to make work.[504.44] [504.44][S03]Nobody wants to listen to two robots talk to each other.[507.46] [507.46][S03]That will fail and be unlistenable after 30 seconds.[512.59] [512.59][S03]You have to master all these very subtle, weird things[515.77] [515.77][S03]that people do in conversation for it to work.[517.76] [517.76][S01] To make it human-like, exactly as you said.[520.052] [520.052][S01]Raiza, I want to come back to those features[521.95] [521.95][S01]a little bit later, to the Audio Overview,[524.42] [524.42][S01]because I also wanted to discuss the origins of NotebookLM.[528.58] [528.58][S01]How did it come about, Raiza?[530.353] [530.353][S02] For one, I think a lot of people[532.27] [532.27][S02]think that NotebookLM is new because of the Audio Overview[535.48] [535.48][S02]feature.[536.0] [536.0][S02]We had such a massive influx of people and people[538.51] [538.51][S02]were like, wow, what is this?[539.93] [539.93][S02]A brand-new thing from Google.[541.4] [541.4][S02]But actually, we've been working on NotebookLM for over a year.[544.49] [544.49][S02]We first announced it at Google I/O[547.24] [547.24][S02]last year, as Project Tailwind.[549.43] [549.43][S02]And before then, we actually had been incubating it inside[553.12] [553.12][S02]of Google Labs.[554.39] [554.39][S02]And it's actually how Steven and I met.[557.38] [557.38][S02]Steven was brought in.[559.288] [559.288][S02]What was your original title, Steven?[560.83] [560.83][S03] I was visiting scholar.[562.42] [562.42][S02] Yeah.[562.69] [562.69][LAUGHS][562.96] [562.96][S03] Yes.[563.9] [563.9][S03]Then I became editorial director.[566.157] [566.157][S02] That's right, he was promoted.[567.99] [567.99][S02]And at the time, Josh Woodward, who now leads Google Labs--[573.18] [573.18][S02]he's the vice president--[574.97] [574.97][S02]told me, he was like, I want you to build a new AI business.[578.85] [578.85][S02]And I thought to myself, what does it[580.47] [580.47][S02]take to actually do that?[581.83] [581.83][S02]But what I'll say is, one of my early inspirations[584.37] [584.37][S02]was just watching Steven work.[586.26] [586.26][S02]Honestly, just understanding how he does, what he does,[589.398] [589.398][S02]I was like, wow, that could be a real superpower if you[591.69] [591.69][S02]could give that to people.[592.92] [592.92][S03] It was a mix of Steven[594.503] [594.503][S03]is abnormal in his research habits,[597.75] [597.75][S03]but maybe we could turn this into a mainstream pursuit[600.0] [600.0][S03]somehow.[600.72] [600.72][S03]Yeah, it was interesting because I had had this long history[605.34] [605.34][S03]writing books, and Josh had read some of those books[608.97] [608.97][S03]and had read some things that I was writing about,[611.28] [611.28][S03]tools for thought, basically, like how do you use software[614.07] [614.07][S03]to help you think and help you develop your ideas and research?[617.05] [617.05][S03]This is the middle of 2022, so language models[619.32] [619.32][S03]were at the top of the list then.[620.94] [620.94][S03]And so he kind of reached out to me and said,[622.96] [622.96][S03]hey, any chance you would want to come to Google[625.68] [625.68][S03]and help build the tool that you have always[628.02] [628.02][S03]wanted to help people learn and organize their ideas,[632.07] [632.07][S03]now built on top of language models?[633.81] [633.81][S03]And what Raiza and I-- kind of right from the beginning--[636.28] [636.28][S03]I think I met Raiza day two at Google.[639.64] [639.64][S03]We were like, let's build something new.[641.51] [641.51][S01] This came about at a time[642.64] [642.64][S01]when large language models were at the top of the agenda.[645.015] [645.015][S01]In those early conversations, how[646.39] [646.39][S01]did you see this as being fundamentally different to just,[649.658] [649.658][S01]I don't know, like uploading a document on Gemini[651.7] [651.7][S01]and getting it to summarize it for you?[653.29] [653.29][S03] From the very beginning--[654.41] [654.41][S03]we call it source grounding.[655.85] [655.85][S03]That's the way we describe it.[657.1] [657.1][S03]You supply the source information[658.48] [658.48][S03]that you want to work with.[659.36] [659.36][S03]It might be the story you're writing.[660.77] [660.77][S03]It might be the book you're researching.[662.437] [662.437][S03]It might be your journals.[663.71] [663.71][S03]It might be the marketing documents you're working on.[665.96] [665.96][S03]And uploading that to the model then creates[668.53] [668.53][S03]a kind of personalized AI that is an expert in the information[672.52] [672.52][S03]that you care about.[673.64] [673.64][S03]And that was not-- no one was talking about that[676.36] [676.36][S03]in the middle of 2022.[678.17] [678.17][S03]So that was like the first thing we built, was like--[680.87] [680.87][S03]I mean, we uploaded part of one of my books,[683.32] [683.32][S03]and I could have this very crude conversation with the model that[687.46] [687.46][S03]was not at all like what you see now in text or with audio.[691.13] [691.13][S03]But you could get a little taste of what[693.67] [693.67][S03]it would be like to have all the ideas you were working with[697.77] [697.77][S03]instead of just talking to an open-ended model that just[700.17] [700.17][S03]had its general knowledge, actually have[701.64] [701.64][S03]that personalized knowledge.[702.807] [702.807][S03]And it was great because it also reduced hallucinations.[706.36] [706.36][S03]It made it more factual.[707.65] [707.65][S03]You could fact-check it.[709.17] [709.17][S03]You could go back and see the original source material.[711.73] [711.73][S03]That's a big part of the whole NotebookLM experience.[714.75] [714.75][S03]That was the beginning of it.[716.02] [716.02][S03]And everything we've done is built on that platform,[718.187] [718.187][S03]and Audio Overviews is just, OK, take that insight of,[720.75] [720.75][S03]I supply my sources and now, I turn it into something else.[723.46] [723.46][S03]In this case, it's an audio conversation.[725.64] [725.64][S01] Because I guess the real key difference[727.08] [727.08][S01]here is that it's very focused on the sources[729.0] [729.0][S01]that you're giving it and anything[731.08] [731.08][S01]that's connected to that rather than just as you[733.08] [733.08][S01]say, this general model.[734.7] [734.7][S02] Yeah, I think that--[736.71] [736.71][S02]I'll say, too, that what we've seen[739.53] [739.53][S02]is I think it's a little bit harder to get started[742.41] [742.41][S02]with this paradigm because it's so new.[744.63] [744.63][S02]The idea that one, you're talking to an AI, two,[748.065] [748.065][S02]you have to bring your own stuff.[749.44] [749.44][S02]So I think there's a little bit of a layer where[750.93] [750.93][S02]it's like, OK, you have to convince somebody[752.763] [752.763][S02]that it's worth doing.[753.88] [753.88][S02]But once you can get somebody over that hump,[756.22] [756.22][S02]it's just massively useful because--[758.5] [758.5][S02]I think about the work that I do every day,[760.61] [760.61][S02]the work Steven does every day, and many people[763.177] [763.177][S02]around the world that work on computers every day,[765.26] [765.26][S02]we are working with very specific sets of information,[769.43] [769.43][S02]shared contexts that we have with others.[772.21] [772.21][S02]We do research, we pull it in, we[774.1] [774.1][S02]want to extract our own insights from it.[776.48] [776.48][S02]I think that's what makes NotebookLM really special[778.87] [778.87][S02]and has made it special from the beginning.[780.77] [780.77][S01] So it does include these text elements too then,[782.84] [782.84][S01]because as you say, the podcast part[784.34] [784.34][S01]is the bit that's most notable.[787.055] [787.055][S02] That's right.[788.18] [788.18][S02]So the podcast thing is the most recent development[792.31] [792.31][S02]in NotebookLM, but we actually launched a year ago,[795.67] [795.67][S02]where it was primarily a chat feature.[798.53] [798.53][S02]So you're chatting with the system using your sources,[802.97] [802.97][S02]and it's always referencing back to exactly what pieces[806.98] [806.98][S02]of your content that it used.[808.238] [808.238][S01] So give me some more mundane examples[810.28] [810.28][S01]of how people are using this, like on a day-to-day level[812.613] [812.613][S01]then, Steven.[814.0] [814.0][S03] Yeah, so we actually[816.19] [816.19][S03]see a huge amount of usage of the product[818.08] [818.08][S03]just with the text features.[819.61] [819.61][S03]And suddenly, you have this amazing resource[823.03] [823.03][S03]that can answer any question about all-- hundreds of pages[826.3] [826.3][S03]of documents.[827.68] [827.68][S03]And in the text version, you get citations and everything.[830.787] [830.787][S03]It's a very scholarly thing, actually,[832.37] [832.37][S03]and you would appreciate it.[833.537] [833.537][S03]You get your answers back and every fact that the model says[837.34] [837.34][S03]has a little inline footnote, and you[839.452] [839.452][S03]can click directly on that footnote[840.91] [840.91][S03]and go and read the original passage.[843.17] [843.17][S03]Writers, journalists, obviously, are using it.[845.33] [845.33][S03]This comes a little bit out of my involvement for the project.[848.21] [848.21][S03]I have one notebook that has thousands and thousands[852.075] [852.075][S03]of quotes from books that I've read over the years,[854.2] [854.2][S03]plus a lot of the text of books that I've written.[856.7] [856.7][S03]And that notebook has basically like my brain[860.95] [860.95][S03]kind of captured in the AI.[862.85] [862.85][S03]And so whenever I work on anything, I[864.935] [864.935][S03]have a new idea for something, I'll go into that notebook[867.31] [867.31][S03]and be like, hey, what do you think about this idea?[870.825] [870.825][S03]And the AI will say, hey, Steven,[872.87] [872.87][S03]you read something related to that seven years ago.[875.78] [875.78][S03]What about this passage?[877.16] [877.16][S03]And so it's a true extension of my memory, so that kind[882.15] [882.15][S03]of stuff.[882.65] [882.65][S03]And the other thing, last thing I'll say,[883.51] [883.51][S03]is we're not training the model on this information.[886.1] [886.1][S03]So your information is secure, it's private.[888.048] [888.048][S03]It's not going to get into the general knowledge of the model[890.59] [890.59][S03]and be used by somebody else.[891.89] [891.89][S03]So you can put private information in there.[894.35] [894.35][S03]And when you put a couple of years of your journal[899.02] [899.02][S03]in a large-context model like this,[902.89] [902.89][S03]you can get these amazing insights[904.96] [904.96][S03]and you can turn them into audio overviews[907.09] [907.09][S03]and listen to two people talk about yourself.[909.44] [909.44][S03]Or you can just be like, what was I thinking about last May?[913.13] [913.13][S03]Give me an overview of all the stuff that was going on.[916.18] [916.18][S03]And 20 seconds later, you'll have this amazing kind[919.06] [919.06][S03]of document of your own life.[920.485] [920.485][S01] Rather than just recalling stuff,[922.36] [922.36][S01]they can actually be insightful, in terms of your own journals.[925.398] [925.398][S02] I would say yes, because I've[927.19] [927.19][S02]used it for that purpose.[928.94] [928.94][S02]And one of the things that I like to ask it after uploading--[932.72] [932.72][S02]I do these weekly journals --is I say, how much have I[936.01] [936.01][S02]changed over time?[937.84] [937.84][S02]And it's really remarkable.[940.55] [940.55][S02]It's been able to pull out for me, really interesting nuances[943.94] [943.94][S02]that I haven't been able to observe about myself.[946.94] [946.94][S02]It's been able to say things like hey,[949.46] [949.46][S02]you tend to associate a lot of negativity[951.68] [951.68][S02]with this particular topic.[953.91] [953.91][S02]You associate a lot of positivity with this topic.[956.4] [956.4][S02]And it's just really interesting because I think, to your earlier[959.45] [959.45][S02]question around the mundane use cases,[962.28] [962.28][S02]I think we see a lot more of those,[965.52] [965.52][S02]which is just people trying to take the work that they're[968.6] [968.6][S02]doing every day.[969.69] [969.69][S02]For example, sales teams use this a lot[971.93] [971.93][S02]to share knowledge with each other.[973.46] [973.46][S02]Makes a lot of sense.[974.43] [974.43][S02]There's a lot of technical, complex changing documentation,[977.61] [977.61][S02]so it's really nice to have an AI partner.[979.65] [979.65][S02]I think that's really different from how a lot of AI systems[983.42] [983.42][S02]work today, right?[984.9] [984.9][S02]I use everything.[987.24] [987.24][S02]I use everything that's out there,[988.71] [988.71][S02]and the prompts that I write are massive.[991.77] [991.77][S02]The first thing that I write is, you are a blah.[994.5] [994.5][S02]This is what we are doing.[996.03] [996.03][S02]Here are the documents that are relevant.[998.22] [998.22][S02]And I think for NotebookLM, this just shortcuts it.[1001.16] [1001.16][S02]It's just a project space.[1002.282] [1002.282][S02]It knows what you're talking about.[1003.74] [1003.74][S02]You can have a conversation forever.[1005.9] [1005.9][S02]It takes up to 25 million words.[1007.82] [1007.82][S02]It's just contextually quite massive.[1010.687] [1010.687][S03] I think one of the things that was interesting[1013.27] [1013.27][S03]and maybe a little bit distinctive about it[1015.062] [1015.062][S03]was so many of the questions about what makes this product[1018.1] [1018.1][S03]work or not work are not so much technological questions[1021.73] [1021.73][S03]as they are editorial stylistic questions.[1023.89] [1023.89][S03]Like, what is the right kind of answer when you get[1026.38] [1026.38][S03]an audio overview that works?[1030.579] [1030.579][S03]What's the style?[1031.52] [1031.52][S03]What's the house style for those conversations?[1034.0] [1034.0][S03]What level should they be pitched at?[1035.66] [1035.66][S03]And those are not technological questions.[1037.73] [1037.73][S03]Those are language questions.[1039.619] [1039.619][S03]And that's the crazy reality of the language-model age,[1043.383] [1043.383][S03]is that all these things that used to be just mostly[1045.55] [1045.55][S03]a question of, let's get the programming right now,[1048.14] [1048.14][S03]become more about the rhetoric of it all.[1050.973] [1050.973][S01] Well, actually, I want[1052.39] [1052.39][S01]to dig into some of the house style a little bit more,[1055.41] [1055.41][S01]I guess.[1055.91] [1055.91][S01]Why did you decide to go into Audio Overview?[1058.43] [1058.43][S01]What was it that inspired that?[1059.99] [1059.99][S01]I mean, there are already quite a lot of podcasts.[1062.17] [1062.17][S01]Let's be honest.[1063.58] [1063.58][S03] When Audio Overviews really began--[1066.02] [1066.02][S03]it was a great example of the lab's structure,[1069.35] [1069.35][S03]I think, really working well because it[1071.98] [1071.98][S03]was another small team inside of labs[1074.35] [1074.35][S03]that were just kind of focused on the audio version of this.[1077.565] [1080.71][S03]And part of the idea of it was not[1082.78] [1082.78][S03]so much to compete with podcasts but rather[1084.99] [1084.99][S03]that there was a whole universe of content[1086.74] [1086.74][S03]that you would never-- the economics of generating[1089.47] [1089.47][S03]a podcast for it would never make any sense.[1091.7] [1091.7][S03]But if you could generate one automatically,[1094.67] [1094.67][S03]you might have five people that would want to listen to it,[1097.132] [1097.132][S03]or one person who would want to listen to it, or 20 people,[1099.59] [1099.59][S03]but not 200,000.[1101.21] [1101.21][S03]And so we want to create a podcast[1104.47] [1104.47][S03]based on our team meetings from the last week[1106.69] [1106.69][S03]so we can review them.[1107.69] [1107.69][S03]That's not going to be a commercial business.[1110.05] [1110.05][S03]No one's going to ask you to host that,[1113.052] [1113.052][S03]but actually, it might be useful for that team.[1115.01] [1115.01][S03]And so they had started developing this thing,[1117.55] [1117.55][S03]and Raiza and I heard it probably[1120.96] [1120.96][S03]in March or April of this year.[1123.88] [1123.88][S03]And as everyone who's heard an audio overview,[1127.17] [1127.17][S03]initially, we're just like, wow, what did I just hear?[1129.97] [1129.97][S03]That was amazing.[1130.93] [1130.93][S03]But we realized pretty early on that part of our mission[1135.03] [1135.03][S03]with NotebookLM was to build a tool that[1137.22] [1137.22][S03]helps people understand things.[1138.7] [1138.7][S03]And suddenly, we were like, oh wait, people really[1141.09] [1141.09][S03]understand, and remember, and pay attention[1144.21] [1144.21][S03]when they hear something in the form of an engaging conversation[1147.75] [1147.75][S03]between two smart people.[1149.4] [1149.4][S03]We've released it internally to Googlers over the summer,[1153.34] [1153.34][S03]and that was, I think, when we started[1155.31] [1155.31][S03]to think this is going to be a hit,[1157.8] [1157.8][S03]because you could just see the delight that people had with it.[1161.1] [1161.1][S03]So while we were surprised that it[1163.86] [1163.86][S03]went quite as crazy as it did, we knew we were on to something.[1167.52] [1167.52][S01] Now, I remember the last season,[1170.01] [1170.01][S01]we got to hear a demo of WaveNet, which, of course, is[1172.62] [1172.62][S01]one of the first AI models to generate this human-like speech.[1176.26] [1176.26][S01]And it was quite impressive back then,[1179.54] [1179.54][S01]but I mean, presumably, there have[1181.22] [1181.22][S01]been technological advancements that have happened since, that[1183.915] [1183.915][S01]have been necessary to make something like Audio Overview[1186.29] [1186.29][S01]possible.[1187.32] [1187.32][S02] I think the underlying model for NotebookLM[1191.09] [1191.09][S02]is Gemini 1.5 Pro, and that just creates, really,[1196.76] [1196.76][S02]to me, incredible content.[1199.5] [1199.5][S02]The voice models, the audio model that we use, that,[1203.0] [1203.0][S02]by itself, is a breakthrough.[1204.795] [1204.795][S02]And I think that's what you're talking about,[1206.67] [1206.67][S02]which is the realism of the human voice,[1209.37] [1209.37][S02]the human-like voices that we hear,[1211.4] [1211.4][S02]and pair that with the approach that we've taken--[1215.04] [1215.04][S02]and Steven can speak more to this, too--[1217.46] [1217.46][S02]of editorializing the content to thinking about,[1220.19] [1220.19][S02]how do we create something really useful[1222.26] [1222.26][S02]and really fun for you that's engaging?[1224.85] [1224.85][S03] Yeah, that's a great segue actually,[1228.03] [1228.03][S03]to something I was going to say, which is about interestingness.[1231.32] [1231.32][S03]So Simon, who's one of the leads on the audio side,[1237.02] [1237.02][S03]he sometimes has a slogan for Audio Overviews, which[1240.57] [1240.57][S03]is make anything interesting.[1242.47] [1242.47][LAUGHS][1243.148] [1243.148][S03]So, like, whatever, make your dissertation interesting.[1245.44] [1245.44][S03]I'm sure it was interesting.[1246.69] [1246.69][S01] It wasn't.[1247.607] [1247.607][LAUGHS][1249.218] [1249.218][S03] And so it's a great example[1251.01] [1251.01][S03]of a convergence of three different technologies[1254.67] [1254.67][S03]or breakthroughs that make something magical happen.[1257.97] [1257.97][S03]Gemini itself, and it can do this with text as well,[1261.76] [1261.76][S03]is incredibly good at pulling out[1263.97] [1263.97][S03]interesting facts or ideas or stories[1266.51] [1266.51][S03]from the material you give it.[1267.76] [1267.76][S03]So I do this all the time.[1269.14] [1269.14][S03]I upload something new and say, tell me[1270.79] [1270.79][S03]the most interesting things from this, just in text.[1272.86] [1272.86][S03]Computers could never do that before.[1274.402] [1274.402][S03]You couldn't command F for interestingness.[1276.37] [1276.37][S03]This was not a search query you could do.[1278.125] [1278.125][S01] But how are you defining it even?[1280.0] [1280.0][S01]I mean, what does it mean?[1281.108] [1281.108][S03] I believe that it comes out[1282.9] [1282.9][S03]of the basic idea behind language models, which[1285.84] [1285.84][S03]is that they're predictive.[1287.1] [1287.1][S03]They're like, given this string of text,[1289.15] [1289.15][S03]I expect the next thing to happen.[1290.71] [1290.71][S03]And so what interestingness is kind of controlled surprise.[1295.733] [1295.733][S03]I thought this was going to be the case,[1297.4] [1297.4][S03]but actually, there's some new information here[1299.358] [1299.358][S03]that I wasn't expecting.[1300.61] [1300.61][S03]And so it makes sense, in a way, that the language models[1304.38] [1304.38][S03]would be good at this because their basic circuitry is[1307.95] [1307.95][S03]prediction.[1308.59] [1308.59][S03]And so they're looking through all this information.[1310.757] [1310.757][S03]Given their training data, what in this information[1312.99] [1312.99][S03]is novel or--[1313.867] [1313.867][S01] Surprising.[1314.825] [1314.825][S03] --defies their expectations.[1316.57] [1316.57][S03]So it's very good at that.[1317.44] [1317.44][S03]So that's an underlying Gemini thing, right?[1319.33] [1319.33][S03]And the hosts of the show are instructed[1321.72] [1321.72][S03]to find the interesting material and present it[1325.95] [1325.95][S03]to the user in an engaging way.[1328.03] [1328.03][S03]So that's one capability.[1329.71] [1329.71][S03]The second thing that is really cool about this[1332.4] [1332.4][S03]is that the instructions take the script that is generated,[1337.45] [1337.45][S03]and they add noise to the script.[1339.73] [1339.73][S03]So they add what are called disfluencies,[1342.1] [1342.1][S03]so all the stammers, and the "likes,"[1344.04] [1344.04][S03]and the interjections that humans actually have when they[1348.21] [1348.21][S03]speak.[1348.73] [1348.73][S03]And it turns out you need that because if you[1351.03] [1351.03][S03]don't have that noise, it sounds too robotic.[1354.653] [1354.653][S01] Mhm.[1355.32] [1355.32][S03] And then finally, there's[1357.028] [1357.028][S03]the audio voices themselves, and what[1358.71] [1358.71][S03]they do is all these subtle things, like in English,[1363.17] [1363.17][S03]speakers will raise their voice a little bit if they're not[1366.168] [1366.168][S03]sure about what they're saying.[1367.46] [1367.46][S03]Or for emphasis, they will slow down what they're saying.[1370.47] [1370.47][S03]All these things that we do natively,[1373.14] [1373.14][S03]we never even think about it.[1374.43] [1374.43][S03]But no computer could do that until now,[1376.86] [1376.86][S03]and that's the part of it that just like lights up.[1379.35] [1379.35][S03]And that's the underlying language vocal model,[1382.14] [1382.14][S03]audio model, that didn't exist a year ago.[1384.935] [1384.935][S01] It's the voice modulation, right?[1386.81] [1386.81][S01]It's like when you--[1389.39] [1389.39][S01]I remember years ago at the BBC, being taught[1394.25] [1394.25][S01]to make content sound engaging.[1396.86] [1396.86][S01]And they give you a copy of "Winnie the Pooh" to read.[1399.56] [1399.56][S01]And then they say, OK, read it as you would a newsreader[1402.89] [1402.89][S01]and you read it very, very flat.[1404.28] [1404.28][S01]And then they say, read it as you would to a child,[1406.07] [1406.07][S01]and you notice, exactly as you say,[1407.61] [1407.61][S01]Steven, that your voice goes up at certain points[1409.89] [1409.89][S01]and it goes down at other points.[1411.66] [1411.66][S01]The range that you have and the speeds completely changes.[1414.33] [1414.33][S01]But you've built all of those aspects into this.[1418.05] [1418.05][S01]I mean, how on Earth do you do that?[1420.655] [1420.655][S03] Yeah, we should make it very clear,[1422.78] [1422.78][S03]we do not build the vocal model.[1424.902] [1424.902][S01] You plural, you plural.[1426.36] [1426.36][S03] We have no idea how it was built,[1428.402] [1428.402][S03]and geniuses inside of Google built that,[1431.54] [1431.54][S03]and we inherited that technology.[1433.11] [1433.11][S03]And we have been running with it and showing[1434.63] [1434.63][S03]how it could be useful, but we did not build it.[1436.64] [1436.64][S03]One of the questions that people have is--[1441.02] [1441.02][S03]it's English only right now, and people[1443.27] [1443.27][S03]are very eager for it to come into different languages.[1446.55] [1446.55][S03]And we are very eager for that, too,[1448.655] [1448.655][S03]because we have a wonderful international audience.[1450.78] [1450.78][S03]But it's not something you can do easily[1452.51] [1452.51][S03]because the intonations and all those little conversational tics[1456.47] [1456.47][S03]are different in every language.[1458.16] [1458.16][S03]And so you can't just be like, change the words into Spanish[1461.15] [1461.15][S03]and press Play.[1462.76] [1462.76][S02] I was just going to add that DeepMind actually[1465.26] [1465.26][S02]has a recent blog post about the audio model,[1469.37] [1469.37][S02]and how it was built, and who built it, and all the research[1472.07] [1472.07][S02]papers underneath it.[1473.4] [1473.4][S02]I think if we could share that, we should.[1476.022] [1476.022][S01] Yeah.[1476.73] [1476.73][S01]No, absolutely.[1477.71] [1477.71][S01]I think one thing that's really noticeable playing around[1480.085] [1480.085][S01]with this is how it is very versatile across different types[1484.56] [1484.56][S01]of data that you give it.[1486.72] [1486.72][S01]And so the way that you're describing[1488.3] [1488.3][S01]this, Steven, is that you're sort[1489.68] [1489.68][S01]of coding in all of the disfluencies,[1492.5] [1492.5][S01]but how do you stop this thing from just[1495.14] [1495.14][S01]sounding like a bunch of cliches every single time?[1497.51] [1497.51][S01]Raiza.[1498.482] [1498.482][S02] I actually think it's[1499.94] [1499.94][S02]hard to get it not to sound like a bunch of cliches every time.[1503.61] [1503.61][S02]I think because of trying to standardize interestingness,[1509.25] [1509.25][S02]that's really, actually quite difficult.[1512.16] [1512.16][S02]And so interestingness tends to sound the same after hearing it[1515.78] [1515.78][S02]enough times.[1517.4] [1517.4][S02]And that's why we actually introduced the first improvement[1520.55] [1520.55][S02]to this particular launch, which is, we're letting users--[1524.4] [1524.4][S02]I call it pass a note to the hosts,[1526.46] [1526.46][S02]where you could slip them a little instruction on, hey, you[1529.35] [1529.35][S02]know what, maybe less of the cliche.[1531.51] [1531.51][S02]Go deeper on this topic.[1533.25] [1533.25][S02]And it will change the way that they talk about whatever content[1537.38] [1537.38][S02]you've given them.[1538.145] [1538.145][S01] Should I imagine this[1539.52] [1539.52][S01]as almost though you have different kind of dials?[1542.32] [1542.32][S01]Like, maybe you turn up the quirky dial,[1544.62] [1544.62][S01]and maybe you turn up the historical-fact dial.[1548.07] [1548.07][S01]Or how can I think of this?[1550.77] [1550.77][S03] Well, imagine one thing[1552.87] [1552.87][S03]that I'm very interested in.[1555.58] [1555.58][S03]What if you could give each of the hosts[1558.87] [1558.87][S03]a different kind of field of expertise?[1562.62] [1562.62][S03]Right now, they basically are kind of interchangeable.[1565.6] [1565.6][S03]They don't have defined perspectives on the world.[1568.38] [1568.38][S03]One takes the lead in the conversation,[1570.6] [1570.6][S03]and we switch back and forth randomly.[1573.81] [1573.81][S03]But what if you were like, OK, I'm a city planner,[1576.51] [1576.51][S03]and I'm working on this design for this new town square.[1580.2] [1580.2][S03]And I want one of them to be an environmental activist,[1582.96] [1582.96][S03]and I want one of them to be an economist.[1585.4] [1585.4][S03]And now, let's have a conversation[1587.79] [1587.79][S03]and let's have a debate.[1589.12] [1589.12][S03]And suddenly, they have different perspectives[1591.295] [1591.295][S03]because one of the things-- this is something[1593.17] [1593.17][S03]I've written about a lot in my books over the years[1595.83] [1595.83][S03]--is that people are more creative, make better decisions,[1599.97] [1599.97][S03]when they have a diverse pool of expertise helping[1603.72] [1603.72][S03]them make the choices or come up with the ideas they're[1606.09] [1606.09][S03]trying to do.[1606.73] [1606.73][S03]And that's also on our roadmap for 2025.[1611.002] [1611.002][S01] Will I actually be able to interact[1612.96] [1612.96][S01]with these hosts in the future?[1614.41] [1614.41][S01]Like, I don't know, interrupt them[1615.9] [1615.9][S01]and join their conversation?[1617.915] [1617.915][S03] Well, we actually[1619.29] [1619.29][S03]showed a version of this at I/O, the big Google Developers[1624.24] [1624.24][S03]Conference where we first rolled out this feature, announced it.[1627.22] [1627.22][S03]And they do their audio-podcast format,[1631.63] [1631.63][S03]and then Josh Woodward, the head of Labs, in the demo,[1634.78] [1634.78][S03]interrupts and says, hey, can you--[1636.39] [1636.39][S03]they're talking about physics.[1638.22] [1638.22][S03]And he's like, hey, can you use a basketball metaphor here,[1641.44] [1641.44][S03]because my son is listening?[1642.93] [1642.93][S03]And they're like, oh, great.[1644.56] [1644.56][S03]OK.[1645.3] [1645.3][S03]Someone called into the show, basically, and they're like,[1648.06] [1648.06][S03]let's do it in a basketball metaphor.[1651.07] [1651.07][S03]So that has been, publicly, part of what we wanted to do,[1655.72] [1655.72][S03]and you can imagine we're very eager to bring that to people.[1660.37] [1660.37][S01] I mean, you paint a really compelling picture.[1663.16] [1663.16][S01]I do also wonder, though, is there the danger[1665.46] [1665.46][S01]here that you could have--[1668.11] [1668.11][S01]picking up on a minor detail in the corpus of text,[1671.34] [1671.34][S01]and then make it into a much bigger thing than it necessarily[1674.42] [1674.42][S01]is?[1674.92] [1674.92][S01]I mean, we're still at the situation[1676.42] [1676.42][S01]where large language models can kind of hallucinate,[1679.78] [1679.78][S01]can not necessarily put the right emphasis[1681.66] [1681.66][S01]on different parts of what it's reporting.[1684.585] [1684.585][S03] The early days of three weeks[1686.46] [1686.46][S03]ago, when we were testing this customization passive note[1689.73] [1689.73][S03]from the producers feature that Raiza's talking about,[1691.98] [1691.98][S03]I uploaded an article I'd written a couple of years ago,[1695.04] [1695.04][S03]and I gave them the instructions to give me[1699.39] [1699.39][S03]relentless criticism of this piece in the style of an insult[1704.19] [1704.19][S03]comic at a roast because, again, they're kind of instructed[1709.26] [1709.26][S03]to be enthusiastic.[1710.56] [1710.56][S03]So I upload this piece, and it was cool.[1713.38] [1713.38][S03]They immediately were like, what is Johnson's problem?[1716.76] [1716.76][S03]Did he even do any research for this piece?[1719.19] [1719.19][S03]But they also kind of reached for a criticism of it that,[1723.78] [1723.78][S03]genuinely--[1724.62] [1724.62][S03]I'm not just saying this because I[1726.037] [1726.037][S03]wrote it and I'm defensive-- was kind of wrong.[1728.112] [1728.112][S03]It kind of misread it a little bit.[1729.57] [1729.57][S03]And I couldn't quite tell whether it[1731.07] [1731.07][S03]was because I'd instructed them to be so extreme[1733.56] [1733.56][S03]or whether they just-- it's almost like,[1735.872] [1735.872][S03]I keep saying this to people.[1737.08] [1737.08][S03]It's like they don't really hallucinate in the way[1739.29] [1739.29][S03]that the first-generation models do.[1741.43] [1741.43][S03]It's just that they sometimes get confused[1743.22] [1743.22][S03]or they misinterpret something in a way that humans do,[1746.37] [1746.37][S03]and their take is a little bit off.[1748.62] [1748.62][S01] Well, what about humor though?[1750.37] [1750.37][S01]I mean, we're talking about all of these different types[1752.25] [1752.25][S01]of examples.[1753.06] [1753.06][S01]Have they ever made you laugh?[1755.03] [1755.03][S02] Yes.[1755.78] [1755.78][S02]Yes.[1756.367] [1756.367][S03] Yes.[1757.2] [1757.2][S02] Actually, I will say[1758.617] [1758.617][S02]that they have made me laugh through the cleverness,[1763.35] [1763.35][S02]and the humor, and the exploration of other people,[1767.02] [1767.02][S02]because I myself--[1768.443] [1768.443][S02]I don't think I could have come up[1769.86] [1769.86][S02]with the funny cases on my own.[1771.76] [1771.76][S02]But just seeing what people have tried[1774.66] [1774.66][S02]in the outside world with the technology,[1777.98] [1777.98][S02]that's been really funny.[1779.64] [1779.64][S02]And somebody uploaded a document to NotebookLM,[1782.6] [1782.6][S02]and the document just said the words "poop" and "fart" in it.[1787.74] [1787.74][S02]And when I saw that that's what it was-- the person[1790.13] [1790.13][S02]posted it on Twitter.[1791.07] [1791.07][S02]They're like, that's all this is.[1792.445] [1792.445][S02]Listen to the podcast.[1793.92] [1793.92][S02]I was like oh dear.[1796.4] [1796.4][S02]What is this about to be?[1797.97] [1797.97][S02]But it was hilarious.[1799.38] [1799.38][S02]It was so good.[1800.43] [1800.43][S02]And the thing that makes it so funny[1803.27] [1803.27][S02]is that there were moments that were truly hilarious,[1806.3] [1806.3][S02]and then it would dip into, but what does it really mean?[1810.455] [1810.455][S02]And it would be thoughtful.[1811.58] [1811.58][S02]It would be bizarre.[1812.95] [1812.95][S02]It would be thought provoking.[1814.2] [1814.2][S02]And I'm like, am I really listening to this?[1816.502] [1816.502][S02]But I took it very seriously.[1817.71] [1817.71][S02]It was great.[1818.335] [1818.335][S01] Yeah, I guess in some ways,[1819.96] [1819.96][S01]though, that's sort of hilarious in the way[1822.56] [1822.56][S01]that the AI is kind of oblivious to how absurd the challenge is,[1827.96] [1827.96][S01]that it's been said.[1829.027] [1829.027][S02] I think on that one, they mentioned,[1831.11] [1831.11][S02]is somebody's trying to trick us into just saying a bunch[1833.84] [1833.84][S02]of "poop" and "fart"?[1835.73] [1835.73][S02]And I was like, I think so.[1838.35] [1838.35][S01] I do also think that the more traditional forms[1841.76] [1841.76][S01]of humor, so not just laughing at how oblivious the AI is,[1845.19] [1845.19][S01]but a lot of that seems to me like it's about the build up[1848.232] [1848.232][S01]and release of tension.[1849.19] [1849.19][S03] Yeah.[1849.51] [1849.51][S01] So, it's the kind of similar thing[1851.427] [1851.427][S01]about you're making a prediction of where you're[1853.485] [1853.485][S01]expecting a sentence to go, and then it[1855.11] [1855.11][S01]goes in a different direction.[1856.5] [1856.5][S01]Is this something that you think that it will[1858.74] [1858.74][S01]be able to do in the future?[1860.19] [1860.19][S01]Because I don't think it's particularly good at it now.[1862.792] [1862.792][S03] I actually had this sense[1864.5] [1864.5][S03]in the early days, the first couple of weeks,[1866.28] [1866.28][S03]really, that it was out--[1867.322] [1867.322][S03]I actually wrote about this briefly--[1869.58] [1869.58][S03]which was that they actually weren't very good at humor.[1872.34] [1872.34][S03]They had banter and they were playful,[1876.14] [1876.14][S03]but they didn't really like crack good jokes[1878.99] [1878.99][S03]or have genuinely funny things.[1880.53] [1880.53][S03]And then it turned out, as Raiza said,[1882.68] [1882.68][S03]that users were able to push them into being genuinely funny.[1886.77] [1886.77][S03]They had to be put in a funny situation, as it were.[1889.23] [1889.23][S03]Like, we've been given this poop-fart document.[1891.27] [1891.27][S03]Another one was a completely coherent-looking scientific[1895.91] [1895.91][S03]paper with charts, and graphs, and published in with footnotes[1899.268] [1899.268][S03]and everything, except that every word in the paper was[1901.56] [1901.56][S03]"chicken," just "chicken," "chicken," "chicken," "chicken,"[1903.37] [1903.37][S03]"chicken," "chicken."[1904.26] [1904.26][S03]And every footnote was "chicken," "chicken," "chicken,"[1905.92] [1905.92][S03]"chicken."[1906.16] [1906.16][S03]All the charts were "chicken."[1907.41] [1907.41][S03]And so they gave them that, and that was the first part where[1909.99] [1909.99][S03]I actually really laughed.[1911.197] [1911.197][S03]They were just like, what is even happening?[1913.03] [1913.03][S03]And they made some funny jokes.[1915.01] [1915.01][S03]And so it's like they have to be prodded[1918.15] [1918.15][S03]into it by an unusual situation, in a weird way.[1921.28] [1921.28][S01] You did mention something there, actually,[1923.53] [1923.53][S01]that I want to pick up on.[1924.613] [1924.613][S01]There are people who have made a criticism of this technology,[1927.36] [1927.36][S01]saying that it's a threat to the podcasting world, that you could[1930.45] [1930.45][S01]be flooding the podcasting world with lots[1932.97] [1932.97][S01]of generic, low-quality AI-generated podcasts.[1937.12] [1937.12][S01]Is there a response that you have to that?[1939.047] [1939.047][S02] What is most interesting and nuanced[1941.13] [1941.13][S02]about it is that what we've found[1943.38] [1943.38][S02]is that people are creating content[1945.18] [1945.18][S02]of things that probably don't have a podcast about it[1948.06] [1948.06][S02]to begin with.[1948.97] [1948.97][S02]It really is--[1950.02] [1950.02][S02]I don't want to say mundane.[1951.79] [1951.79][S02]But it really is things that nobody is going[1954.06] [1954.06][S02]to make a whole show about.[1956.007] [1956.007][S02]And I think that is interesting.[1957.34] [1957.34][S02]I think we're putting power in people's hands[1960.39] [1960.39][S02]to create content that they want that they ordinarily[1963.63] [1963.63][S02]wouldn't have access to.[1964.75] [1964.75][S02]The second piece of this around the low-quality content,[1970.71] [1970.71][S02]I would say that most of the content that I have heard,[1975.75] [1975.75][S02]ones on the internet, just people posting on the Discord,[1979.89] [1979.89][S02]the quality is quite high.[1981.55] [1981.55][S02]I think on the third note, all of the generations[1986.01] [1986.01][S02]from NotebookLM are also watermarked with SynthID,[1989.13] [1989.13][S02]and so we've taken a very responsible and cautious[1992.61] [1992.61][S02]approach to making sure that as we create the machinery,[1997.02] [1997.02][S02]or as we launch machinery, where you can create[2000.29] [2000.29][S02]audio outputs that are very human-like,[2002.96] [2002.96][S02]we want to make sure that we approach that with watermarking.[2005.505] [2005.505][S03] One of the other things[2007.13] [2007.13][S03]that's interesting here that I think[2008.63] [2008.63][S03]you're getting at a little bit in this line of questioning[2011.66] [2011.66][S03]is, we are personifying these people.[2016.82] [2016.82][S03]They do sound human, and we do all these things[2018.8] [2018.8][S03]to make them sound human.[2019.842] [2019.842][S03]And the interesting thing about this[2021.98] [2021.98][S03]is actually the philosophy that we've[2024.32] [2024.32][S03]had up until Audio Overviews with the product,[2028.1] [2028.1][S03]was in the text version of NotebookLM,[2030.5] [2030.5][S03]it actually does not try to sound particularly human.[2034.23] [2034.23][S03]It's very kind of factual, and it[2036.77] [2036.77][S03]doesn't try to be your friend on some level.[2038.792] [2038.792][S01] Yeah, it's quite cold almost.[2040.5] [2040.5][S03] Yeah, it's almost cold.[2042.125] [2042.125][S03]And that was kind of a bit of the idea of the house style[2046.31] [2046.31][S03]was that, but you can't do that with voice.[2049.469] [2049.469][S03]That's the thing that became very clear the second we first[2053.27] [2053.27][S03]heard these.[2053.85] [2053.85][S03]It's like, you can't say, convey this through a conversation,[2057.78] [2057.78][S03]but don't sound human.[2059.13] [2059.13][S03]Don't pretend to be a person.[2060.6] [2060.6][S03]There's no place where the human ear will tolerate that.[2064.177] [2064.177][S01] I do wonder about that, though,[2065.969] [2065.969][S01]because I mean, in that way, you are, as you say,[2068.489] [2068.489][S01]leaning in a different direction to--[2070.425] [2070.425][S01]I mean, lots of the other conversations[2072.05] [2072.05][S01]that I've had with Google DeepMind about how[2074.15] [2074.15][S01]you should try and avoid anthropomorphization.[2076.199] [2076.199][S01]You should avoid trying to think of them as "they."[2078.429] [2078.429][S01]We've been describing the podcast host as "they"[2080.429] [2080.429][S01]the entire conversation.[2081.86] [2081.86][S01]I mean, are there dangers or concerns[2084.31] [2084.31][S01]that are associated with anthropomorphization[2086.53] [2086.53][S01]of these characters?[2087.87] [2087.87][S02] I think that by personifying them[2090.1] [2090.1][S02]to a certain extent in the way that we have, like[2092.35] [2092.35][S02]adding texture to the way that they describe things,[2095.9] [2095.9][S02]making them sound more human-like,[2097.63] [2097.63][S02]I think it's a way to make information easier to consume[2101.26] [2101.26][S02]and easier to make something more useful.[2104.51] [2104.51][S02]And I think that the reality is that we probably[2108.28] [2108.28][S02]shouldn't resist these types of approaches[2111.79] [2111.79][S02]if we believe that there is enough value associated[2114.48] [2114.48][S02]with them.[2114.98] [2114.98][S02]And I really do.[2116.27] [2116.27][S02]I really think that--[2117.7] [2117.7][S02]I've seen-- I don't know if you've seen on TikTok --all[2120.88] [2120.88][S02]of these people uploading their study materials,[2123.01] [2123.01][S02]and they're like, wow, I can study so much faster.[2126.59] [2126.59][S02]I think about the cases like that where I'm like,[2128.87] [2128.87][S02]are these people being harmed?[2130.64] [2130.64][S02]What is the actual danger?[2132.11] [2132.11][S02]And I'm not saying this to be like, well, clearly, right?[2134.93] [2134.93][S02]It's good for society.[2136.13] [2136.13][S02]But I really am thinking, what are they losing[2138.91] [2138.91][S02]as part of this experience?[2140.24] [2140.24][S02]And I think that it's less about the personification[2144.1] [2144.1][S02]or the anthropomorphization of the hosts themselves[2147.49] [2147.49][S02]and more about, OK, what did you lose by listening[2150.46] [2150.46][S02]instead of reading?[2151.58] [2151.58][S02]Maybe that's it.[2152.66] [2152.66][S03] Yeah, and that's a great point, Raiza.[2154.91] [2154.91][S03]And the other thing that I would add[2156.55] [2156.55][S03]to on that is it turns out to be a very powerful way to learn[2160.21] [2160.21][S03]and to understand is through dialogue,[2162.22] [2162.22][S03]and through asking follow-up questions,[2164.2] [2164.2][S03]and steering the focus towards the things[2167.26] [2167.26][S03]that you need to in a complex body of work.[2170.26] [2170.26][S03]But that kind of dialogue, if you[2172.9] [2172.9][S03]wanted to have a conversation about a book[2176.38] [2176.38][S03]and really engage with it, most people[2180.527] [2180.527][S03]don't have access to the author of the book.[2182.36] [2182.36][S03]Most people don't have access to an expert tutor that understands[2185.068] [2185.068][S03]the complexities of the book.[2186.64] [2186.64][S03]But now, with AI, those kinds of conversational explorations[2190.81] [2190.81][S03]are possible.[2192.16] [2192.16][S01] It's kind of a much more ancient way[2194.2] [2194.2][S01]to explore things, exactly as you describe.[2197.03] [2197.03][S01]I do wonder, though, I mean, you're talking about here,[2199.71] [2199.71][S01]people don't have access to the author,[2201.75] [2201.75][S01]but what's to stop somebody from uploading a book where,[2205.557] [2205.557][S01]actually, you really don't want them to have[2207.39] [2207.39][S01]a conversation with the author?[2208.682] [2208.682][S01]I'm thinking here like putting in "Mein Kampf"[2211.7] [2211.7][S01]or the "Anarchist Cookbook."[2213.318] [2213.318][S03] Yeah, I mean, there's[2214.86] [2214.86][S03]a kind of underlying safety layer[2216.54] [2216.54][S03]that Google spent a lot of time working on,[2218.677] [2218.677][S03]DeepMind and spent a lot of time working on.[2220.51] [2220.51][S03]So if there are obviously offensive, dangerous things,[2223.26] [2223.26][S03]that, you can catch.[2225.96] [2225.96][S03]The trickier thing is, what happens in terms of politics?[2228.695] [2228.695][S03]So if you upload something that's[2230.07] [2230.07][S03]within the bounds of conventional political[2232.32] [2232.32][S03]discussion, but it may be more right wing or more left wing,[2234.9] [2234.9][S03]how should the host respond to that?[2236.83] [2236.83][S03]And so we specifically included instructions that say, listen,[2239.62] [2239.62][S03]if it feels political, then you should[2242.97] [2242.97][S03]adopt the attitude of hey, we're not taking sides in this.[2246.005] [2246.005][S03]We are just going to have a conversation about what[2248.13] [2248.13][S03]this document says, and we're not going to endorse it[2250.68] [2250.68][S03]or critique it in that way.[2252.01] [2252.01][S03]And we figured that was the best compromise[2254.58] [2254.58][S03]for those kind of complicated political stances.[2257.243] [2257.243][S02] I think there's also[2258.66] [2258.66][S02]the interesting sort of line, where, I think, there's[2262.85] [2262.85][S02]the safety concern and then I think[2264.81] [2264.81][S02]there's censorship concern.[2266.68] [2266.68][S02]And actually, in the early days, we ran into this a lot[2270.0] [2270.0][S02]before the safety filters were much more sophisticated, where--[2274.02] [2274.02][S02]people study difficult topics, people[2276.84] [2276.84][S02]study things that happened in history that have quite[2279.24] [2279.24][S02]a bit of violence, racism.[2282.06] [2282.06][S02]These are topics that are fraught.[2284.28] [2284.28][S02]But I think it would be wrong to create a tool that blocks[2289.5] [2289.5][S02]content generically without a thought around the intent[2293.31] [2293.31][S02]of the user so that we're not allowing users[2296.07] [2296.07][S02]to create harmful content, but at the same time, if--[2300.06] [2300.06][S02]most of our users, especially in the beginning,[2302.65] [2302.65][S02]were learners, educators.[2303.76] [2303.76][S02]Like if you're studying history, you[2306.84] [2306.84][S02]are definitely going to run into a safety filter.[2309.23] [2309.23][S03] Well, that was my problem.[2310.98] [2310.98][S03]The last book that I wrote was-- actually,[2312.87] [2312.87][S03]you mentioned "The Anarchist Cookbook."[2314.847] [2314.847][S03]Part of it is about the history of anarchism[2316.68] [2316.68][S03]and the kind of roots of terrorism in the early anarchist[2319.5] [2319.5][S03]world.[2320.3] [2320.3][S03]And so I was using NotebookLM to help me research that book[2323.658] [2323.658][S03]because I was writing it.[2324.7] [2324.7][S03]And it was constantly like, I'm sorry,[2326.24] [2326.24][S03]I can't answer that question because you are obviously[2327.94] [2327.94][S03]a terrorist, Steven.[2328.78] [2328.78][S03]And I'm like, no, no.[2329.735] [2329.735][S01] You're definitely on a list[2331.36] [2331.36][S01]somewhere, Steven, aren't ya?[2332.87] [2332.87][LAUGHS][2333.37] [2333.37][S02] That's right.[2334.36] [2334.36][S03] Maybe I still have a job.[2336.068] [2336.068][LAUGHS][2336.7] [2336.7][S01] There is also this question about personal data.[2339.4] [2339.4][S01]I know that this is something that[2340.817] [2340.817][S01]has been really subject to a lot of discussion[2342.955] [2342.955][S01]with large language models and people uploading documents to it[2345.58] [2345.58][S01]and being concerned about it, kind[2346.997] [2346.997][S01]of feeding into the next generation of models.[2349.48] [2349.48][S01]So how do you make sure, in NotebookLM,[2352.24] [2352.24][S01]as you said, that the information that you upload[2355.72] [2355.72][S01]can be private and remain so?[2358.33] [2358.33][S03] Yeah, so this actually[2360.07] [2360.07][S03]is an opportunity to explain something[2361.37] [2361.37][S03]that I think is really important here,[2362.99] [2362.99][S03]which is the idea of the context window or the model.[2366.34] [2366.34][S03]So a context window is effectively[2368.68] [2368.68][S03]like the short-term memory of a language model.[2371.33] [2371.33][S03]The long term memory is like its training data,[2373.37] [2373.37][S03]like its general knowledge of the world.[2375.29] [2375.29][S03]And the context is the stuff you put in with your query[2379.09] [2379.09][S03]when you ask a question.[2380.65] [2380.65][S03]And anything in the context window is transitory.[2384.288] [2384.288][S03]The second you close your session, it disappears.[2386.33] [2386.33][S03]It gets wiped from the memory of the model.[2388.122] [2388.122][S03]What that also means is that's why it's private.[2390.94] [2390.94][S03]We're not training the model on your information.[2394.46] [2394.46][S03]All we're doing is putting it in the short-term memory[2396.71] [2396.71][S03]in the model, letting the model answer questions.[2399.17] [2399.17][S03]And then when you close the session,[2400.67] [2400.67][S03]it's like the model has completely forgotten anything[2402.61] [2402.61][S03]that you've given to it.[2403.79] [2403.79][S01] So in terms of the future of this--[2406.13] [2406.13][S01]I mean, this is still quite a young product.[2408.61] [2408.61][S01]What are the things are you hoping to include on it?[2411.165] [2411.165][S02] I think we've seen so much excitement[2413.29] [2413.29][S02]about the audio feature, so I think[2415.57] [2415.57][S02]we can definitely commit to that to being on the future roadmap.[2418.77] [2418.77][S02]I think it's alluded to more controls, more voices, more[2421.79] [2421.79][S02]personas, more languages.[2424.59] [2424.59][S02]I think that's just such an exciting horizon for us.[2426.992] [2426.992][S03] The one that I'm so[2428.45] [2428.45][S03]excited to think about, which we've just started[2431.21] [2431.21][S03]to scratch the surface of, is--[2433.04] [2433.04][S03]there's a lot of tools for asking questions and listening[2436.07] [2436.07][S03]to explanations of things, but what[2438.68] [2438.68][S03]about writing with these sources at your disposal?[2443.9] [2443.9][S03]How do we write in a source grounded environment?[2447.02] [2447.02][S03]And so just as a writer myself, I[2449.92] [2449.92][S03]think that that's going to be an amazing thing.[2452.43] [2452.43][S03]So we have some really, really cool things in the works.[2455.96] [2455.96][S01] I do also wonder about different modalities.[2458.43] [2458.43][S01]I mean, you've gone to audio, but, presumably, you[2461.84] [2461.84][S01]could go to video at some point too.[2464.12] [2464.12][S02] Yeah, and, actually, there's a fun idea[2467.33] [2467.33][S02]we have for video, which is like--[2468.96] [2468.96][S02]we're not talking about fully generative video yet,[2471.84] [2471.84][S02]but imagine if you could do even something really basic.[2474.75] [2474.75][S02]You upload these slide decks, they have charts,[2477.39] [2477.39][S02]they have diagrams, you have PDFs of papers.[2480.06] [2480.06][S02]Just take the content that's already there.[2482.16] [2482.16][S02]And NotebookLM is already incredible at this[2484.43] [2484.43][S02]because of our citations model.[2486.77] [2486.77][S02]The fact that we know exactly where every piece of the answer[2492.8] [2492.8][S02]comes from--[2494.15] [2494.15][S02]we use it to generate audio overviews,[2496.23] [2496.23][S02]we use it to generate textual answers.[2498.27] [2498.27][S02]I think it wouldn't be that big of a leap[2500.0] [2500.0][S02]to generate short videos using your own content.[2502.848] [2502.848][S01] I do really like, Steven,[2504.39] [2504.39][S01]how you're describing this often as the thing that you[2507.29] [2507.29][S01]use to make the podcast that nobody else would want to make.[2510.0] [2510.0][S01]But the point here, I guess, is that you're not[2514.4] [2514.4][S01]trying to replace all podcasts.[2516.18] [2516.18][S01]There are presumably things that you expect NotebookLM will never[2519.53] [2519.53][S01]be able to do.[2520.38] [2520.38][S03] Yeah, people, I think,[2522.44] [2522.44][S03]will generally always prefer to hear two actual humans talking[2525.86] [2525.86][S03]about a topic.[2526.62] [2526.62][S03]If there is economics or passion enough[2531.47] [2531.47][S03]to generate a podcast on a topic,[2533.13] [2533.13][S03]humans actually talking to each other will be the choice.[2536.16] [2536.16][S03]It's just turns out that there's this vast,[2539.37] [2539.37][S03]uncharted territory that just wasn't-- no one ever thought[2542.57] [2542.57][S03]about making a podcast based on the family trip to Alaska,[2547.74] [2547.74][S03][LAUGHS] because it just didn't make sense to rent a studio[2552.32] [2552.32][S03]to do that.[2553.11] [2553.11][S03]But now, you can just take everybody's journal entries[2555.62] [2555.62][S03]and photos and upload it to NotebookLM,[2557.54] [2557.54][S03]and you can have a podcast based on your family trip.[2559.86] [2559.86][S03]And so I think that's where it turns out[2562.35] [2562.35][S03]there's just all this untapped kind of blank space on the map[2566.84] [2566.84][S03]that we've just started to explore.[2570.062] [2570.062][S01] Do you think that there[2571.52] [2571.52][S01]are elements of, like, human-content creation[2574.88] [2574.88][S01]that are really hard to capture with AI,[2578.45] [2578.45][S01]or the AI will maybe never be able to capture?[2581.5] [2581.5][S03] Yeah, that's the thing[2583.37] [2583.37][S03]we're trying to figure out.[2585.38] [2585.38][S03]I mean, the one idea I think that I'm really interested in[2590.86] [2590.86][S03]is like, how capable are these models[2594.52] [2594.52][S03]at thinking and developing ideas that are really long form?[2599.8] [2599.8][S03]So book writing-- so when you're coming up with the idea[2603.31] [2603.31][S03]for a book, you're really thinking it's-- one,[2606.22] [2606.22][S03]it's an incredibly long-term process and you're thinking[2608.65] [2608.65][S03]about a presentation of information that's going to go[2611.92] [2611.92][S03]on for 300 pages.[2613.218] [2613.218][S03]It's going to involve all this complexity, all this narrative[2615.76] [2615.76][S03]complexity.[2616.73] [2616.73][S03]And you couldn't approach that all[2619.237] [2619.237][S03]with a language model right now.[2620.57] [2620.57][S03]You could work on little bits of it.[2622.24] [2622.24][S03]You could say, OK, I'm trying to set up this scene[2624.04] [2624.04][S03]or I'm trying to figure out what the narrative should be,[2626.415] [2626.415][S03]but you can't actually imagine the whole thing.[2630.49] [2630.49][S03]That, right now, is just a human-exclusive capability.[2634.22] [2634.22][S03]And I think it will be for a long time, and it may always be.[2637.94] [2637.94][S03]But who knows where we're going to end up?[2639.88] [2639.88][S01] Both the wood and the trees simultaneously.[2642.26] [2642.26][S03] Yeah.[2643.135] [2643.135][S03]Yeah, and I think they're the kind of seeds of that.[2647.63] [2647.63][S03]There's some promising signals, but people[2651.087] [2651.087][S03]who write books for a living, I think,[2652.67] [2652.67][S03]can feel confident that they will[2654.078] [2654.078][S03]continue to be able to do that.[2655.37] [2655.37][S01] Yeah, although writing books for a living[2657.578] [2657.578][S01]is one of the most torturous professions there is.[2659.91] [2659.91][LAUGHS][2660.81] [2660.81][S01]As someone who's trying to write one at the moment,[2662.935] [2662.935][S01]I want you guys to hurry up, please.[2664.97] [2664.97][S01]Well, thank you both for joining me.[2666.75] [2666.75][S01]That was a really, really fascinating discussion.[2669.06] [2669.06][S01]Appreciate it.[2670.07] [2670.07][S03] Thanks for having us.[2671.16] [2671.16][S02] Thank you.[2672.16] [2672.16][S02]Thanks for having us.[2673.56] [2673.56][S01] You know, I think there's actually[2675.477] [2675.477][S01]something quite heartwarming about the way[2677.64] [2677.64][S01]that NotebookLM has captured people's imagination,[2680.137] [2680.137][S01]because on the one hand, you've got this technology that[2682.47] [2682.47][S01]is operating at the absolute cutting edge of what[2685.5] [2685.5][S01]is possible with some of the most sophisticated AI models[2688.92] [2688.92][S01]out there.[2689.77] [2689.77][S01]And it's something that's designed[2691.71] [2691.71][S01]to deal with this very modern problem about how we are often[2695.76] [2695.76][S01]overwhelmed with having to process[2697.53] [2697.53][S01]these large amounts of, often, quite dense and maybe[2701.1] [2701.1][S01]quite-boring information.[2702.87] [2702.87][S01]And they've hit upon a solution that is so innately human, so[2709.08] [2709.08][S01]ancient and appealing, the idea of listening[2712.83] [2712.83][S01]in to a conversation between two excitable and interested people.[2717.61] [2717.61][S01]And, of course, the fastest way to make[2720.09] [2720.09][S01]a human prick up their ears and pay attention is through gossip.[2724.47] [2724.47][S01]And this is like sitting around a fire[2727.29] [2727.29][S01]while an AI uses that very trick to help[2730.95] [2730.95][S01]you digest 25 pages of a snorefest-lecture series.[2735.58] [2735.58][S01]I mean, put it this way.[2736.78] [2736.78][S01]If it can make my PhD thesis sound interesting,[2739.34] [2739.34][S01]then this has the potential to be quite a powerful tool.[2742.33] [2742.33][S01]You have been listening to "Google DeepMind-- the Podcast,"[2745.12] [2745.12][S01]with me, Professor Hannah Fry.[2746.68] [2746.68][S01]If you enjoyed that episode, then[2748.3] [2748.3][S01]do subscribe to our YouTube channel.[2750.64] [2750.64][S01]And you can also find us on your favorite podcast platform.[2753.95] [2753.95][S01]And of course, we have got plenty more episodes[2756.49] [2756.49][S01]on a whole range of topics to come, so do check those out too.[2760.94] [2760.94][S01]See you next time.[2762.25] [2762.25][MUSIC PLAYING][2765.3]