[5.439][S01] Artificial intelligence is slowly appearing[8.208] [8.208][S01]in every aspect of our modern lives.[10.878] [10.878][S01]It’s in our smart phones, our central heating,[13.847] [13.847][S01]on our sideboards and in our cars.[17.084] [17.084][S01]But what about artificial general intelligence?[20.354] [20.354][S01]That is the real quest.[22.656] [22.656][S01]The aim to build an agent - an algorithm -[25.759] [25.759][S01]that can learn to solve any problem from scratch without being taught how.[31.365] [34.968][S01]Welcome to Deep Mind - the podcast. I’m Hannah Fry,[39.706] [39.706][S01]I’m a mathematician who has worked with algorithms for almost a decade.[44.077] [45.012][S01]In this series of podcasts,[46.98] [46.98][S01]we’re following the fast moving story of artificial intelligence.[51.852] [51.852][S01]For the past 12 months we’ve been tracking the latest work[55.088] [55.088][S01]of scientists, researchers and engineers at DeepMind in London.[60.194] [60.194][S01]We’re looking at how they’re approaching the science of AI[63.864] [63.864][S01]and some of the tricky decisions[65.566] [65.566][S01]the whole field is wrestling with at the moment.[68.836] [68.836][S01]So whether you want to know more about where technology is headed,[73.106] [73.106][S01]or want to be inspired on your own AI journey,[76.243] [76.243][S01]then you’ve come to the right place.[78.345] [79.479][S01]Now in the last episode we were talking about[82.082] [82.082][S01]how pitting artificial intelligence[84.384] [84.384][S01]against world class players in the game of Chess,[88.055] [88.055][S01]and the game of Go,[89.59] [89.59][S01]is about much more than just showing off what a computer can do.[93.66] [93.66][S01]Human players can learn from how the AI plays[97.097] [97.097][S01]and improve their own play as a result. And there’s also a bigger picture -[102.336] [102.336][S01]the world of games provides the perfect mini universe[105.772] [105.772][S01]to try out everything we want our artificial intelligence to do.[109.81] [110.577][S01]But intelligence is much more than just championing raw logic,[114.548] [114.548][S01]intelligence requires other skills like the ability to collaborate.[120.053] [121.221][S01]I want to introduce research scientist Max Jaderburg.[124.691] [124.691][S01]Max and his colleagues are trying to work out how to train agents[128.629] [128.629][S01]to work together as a team.[130.531] [130.531][S01]02:11[131.732] [131.732][S03] So imagine it’s a few decades in the future -[133.967] [133.967][S03]we have all these AI systems out in the world doing different things[138.071] [138.071][S03]but they maybe have never seen each other before.[139.84] [139.84][S03]There’s thousands of these things, hundreds of thousands,[142.242] [142.242][S03]each have their own objectives[143.41] [143.41][S03]but somehow they have to cooperate and compete in a sensible way[147.214] [147.214][S03]and in a very ad hoc way, in a way that they’ve never seen each other before.[150.517] [150.517][S01] Humans are really good at this when we want to be, anyway.[154.655] [154.655][S01]Even when we haven’t encountered another person before,[157.09] [157.09][S01]we still know how to understand their intentions[160.327] [160.327][S01]and how to interact with them.[162.529] [162.529][S01]Our agents of the future need to be able to do the same thing with each other.[167.034] [167.034][S03] We already have things like Google Home[169.469] [169.469][S03]and these sort of smart devices out there.[171.305] [171.305][S03]We probably have more and more of those and you can imagine them[174.208] [174.208][S03]having to interact and work with each other[177.644] [177.644][S03]and one device may not have ever seen another device before,[180.781] [180.781][S03]but they still somehow have to interact and get things done for you.[183.951] [183.951][S01] Are we talking about like your Google Home[187.287] [187.287][S01]and your dishwasher here - this kind of stuff or. -[189.556] [189.556][S03] Yeah, potentially. You know your dishwasher[191.291] [191.291][S03]might want to actually go on its cleaning cycle[193.627] [193.627][S03]but Google Home wants it to you know clean all the dishes and so [laughs][197.631] [197.631][S03]what’s best for you as a person - I don’t know?[199.566] [199.566][S01] And who gets to decide? [S03] Who gets to decide?[201.502] [201.502][S01] Who rules supreme - your dishwasher or your Google assistant?[204.805] [204.805][S03] yeah. I don’t know.[206.106] [206.106][S01] There’s an important distinction here.[208.141] [208.141][S01]If you’ve got a smart light bulb that you can program to come on[211.278] [211.278][S01]at six o’clock in the evening, that is an algorithm.[215.382] [215.382][S01]If you’ve got one that can learn your preferences,[217.584] [217.584][S01]that can understand when you like the lights to be dimmed,[221.121] [221.121][S01]what kind of mood lighting you like when you’re reading, that is AI.[226.527] [226.527][S01]But as we switch away from building things that do rigid, pre-decided tasks,[231.665] [231.665][S01]we are asking our technology to read the situation and react[235.435] [235.435][S01]to what’s going on around it.[237.604] [237.604][S01]And in the long-term, that’s going to require collaboration.[242.276] [242.276][S01]So in the spirit of trying things out in a toy universe,[246.079] [246.079][S01]the team at DeepMind have been trying to find inspiration[248.749] [248.749][S01]in another kind of game.[250.784] [250.784][S01]One taken straight from the school playground.[253.921] [256.49][S01] this is Capture The Flag - you know the deal -[259.993] [259.993][S01]the first team to steal the Flag of their opponent[262.396] [262.396][S01]and bring it back to their own base woods.[264.998] [264.998][S01]You get tagged by the opposition, then you’re out of the game.[269.069] [269.069][S01]Oh, come on! Don’t cry.[271.405] [271.405][S01]Max dropped whole populations of AI agents[274.708] [274.708][S01]into a digital version of the game.[277.144] [277.144][S03] This is an onscreen version,[278.345] [278.345][S03]you sort of you just see your first person point of view,[281.281] [281.281][S03]so you have to sort of look around and move through this 3D world[286.353] [286.353][S03]from your own first person perspective, but interact with these other things[289.156] [289.156][S03]which suit their own first person perspective.[291.525] [291.525][S03]So here there’s no centralized entity or being that can see everything.[295.462] [295.462][S01] No army commander. [S03] No army commander.[296.997] [296.997][S03]Every player acts independently.[299.166] [299.166][S03]They only see their own observation and the way that we train[302.569] [302.569][S03]these things we actually train whole populations of teammates -[305.706] [305.706][S03]let’s say 30 agents in parallel[308.075] [308.075][S03]and they are all playing with and against each other.[311.478] [313.146][S01] Rather than just creating a single agent,[315.916] [315.916][S01]Max and his team build an entire classroom of them - 30 in total.[321.255] [321.255][S01]And for each round of the game, he randomly selects[324.057] [324.057][S01]a few of the agents from the class to play together on the team.[328.295] [328.295][S01]By doing this thousands and thousands of times,[331.298] [331.298][S01]each agent will learn from their own experience,[333.834] [333.834][S01]but because they are playing with each other too -[336.203] [336.203][S01]with their classmates as it were - they have to learn to interact[339.806] [339.806][S01]with someone who’s different from themselves.[342.075] [342.075][S03] The problem is when we start it’s actually just very random.[344.578] [344.578][S01] yeah.[344.878] [344.878][S03] They’re just bouncing about the place.[347.047] [347.047][S03]Without a clue [Hannah laughs][348.315] [348.315][S03]and then one of them will discover something[350.951] [350.951][S03]and will start actually let’s say taking control of the flag[355.189] [355.189][S03]and actually scoring points,[356.924] [356.924][S03]and at that point there’s evolutionary pressure on this population.[360.494] [364.865][S01] And here’s the clever bit - Max and his team[367.434] [367.434][S01]aren’t just letting the agents in the classroom play on and on forever,[371.371] [371.371][S01]they’re also using something called a genetic algorithm.[375.242] [375.242][S01]A way to make sure the whole culture of the population of agents evolves.[379.98] [379.98][S03] So actually some of the weaker ones[382.015] [382.015][S03]will be removed from this population.[383.417] [383.417][S01] So it’s almost like you’re making that population of 30 have children.[387.955] [387.955][S03] Yeah, absolutely.[388.488] [388.488][S01] You’re sort of breeding them together.[389.656] [389.656][S03] yeah.[391.358] [393.46][S01] The original classroom of agents breeds together[396.663] [396.663][S01]and have kids of their own.[398.799] [398.799][S01]And as you go down the generations, the strongest traits survive.[403.67] [403.67][S03] But unlike human children, when an agent has children[406.974] [406.974][S03]in this set-up, they inherit everything,[408.942] [408.942][S03]they inherit the knowledge that’s been gained from their parent.[412.045] [412.045][S01] But you’re mixing up the characteristics[414.114] [414.114][S01]as you go from one generation to the next.[415.649] [415.649][S03] Yeah, so this agent has to learn to play[417.551] [417.551][S03]a 5 minute game of Capture The Flag - which is really -[420.587] [420.587][S03]you play 5 minutes, you do thousands of actions,[423.056] [423.056][S03]and you just get a win or a loss whether you have won or lost the game.[426.56] [426.56][S03]Somehow we have to learn what to do with that,[429.296] [429.296][S03]and so to help bridge that problem, we have this idea of internal rewards[433.133] [433.133][S03]where the events in the game such as picking up a flag,[437.738] [437.738][S03]or dropping a flag, or your teammate tagging an opponent[441.742] [441.742][S03]or an opponent tagging you, all these sort of things,[445.612] [445.612][S03]and we allow the agents to individually evolve their own internal rewards,[450.851] [450.851][S03]which is the reward they have assigned to each one of these events.[453.453] [453.453][S01] So some agents are going to care a great deal[455.756] [455.756]about grabbing hold of the flag - [S03] Yeah, exactly -[457.791] [457.791][S01] and other agents are going to care a lot about teammate tagging someone.[461.328] [461.328][S03] Yeah.[463.03] [463.53][S01] This kind of evolutionary group training means[466.4] [466.4][S01]that they can assume different roles.[468.969] [468.969][S01]Producing better results for stealing a flag.[472.306] [472.306][S01]And with a bit of practice, after a few thousand rounds say,[475.776] [475.776][S01]teams of agents become really rather good at this game.[480.48] [481.048][S03] They absolutely smashed it.[482.049] [482.049][S03]And the great thing about training an agent[484.418] [484.418][S03]in this manner is that the robust - yes they can play themselves[487.521] [487.521][S03]but they can play other agents that have been trained[490.023] [490.023][S03]in completely different regimes, they can also play these in-game bots[494.394] [494.394][S03]which are sort of these hard coded bots that shift with the game,[497.598] [497.598][S03]but most interestingly, they can also play with people -[500.167] [500.167][S03]so you can drop people into these games[502.402] [502.402][S03]and have you know an AI teammate or AI opponents.[505.873] [505.873][S01] What was it actually like to play with an agent then - do they -[509.71] [509.71][S01]does it feel like they are guessing what you are going to do[513.447] [513.447][S01]as well as doing their own thing?[515.415] [515.415][S03] it feels less like they are guessing what you are doing[517.417] [517.417][S03]and more like they completely ignore you and they are very ruthless.[521.021] [521.021][S03]Humans pay a lot of attention to other humans -[523.59] [523.59][S03]even in game scenarios like humans will fixate[525.559] [525.559][S03]on the other players of the game,[526.894] [526.894][S03]but these agents have been trained completely unbiased[529.763] [529.763][S03]without these sort of human biases -[532.165] [532.165][S03]your opponent will run right past you and not even try and tag you[535.402] [535.402][S03]because they’re so fixated on actually getting the flag as quickly as possible[538.605] [538.605][S03]because that’s what’s going to maximize their number of flag captures[541.475] [541.475][S03]and win them the game.[542.709] [542.709][S03]Things that really annoying human players would do.[545.279] [546.713][S01] There’s a kind of magic going on here.[549.383] [549.383][S01]Initially researchers are working on these agents[551.718] [551.718][S01]trying to see a way through the muddle.[554.087] [554.087][S01]Then there is that breakthrough moment when the agent gets it.[558.025] [558.025][S01]When they start to behave like you think they should.[561.228] [562.329][S01] Let me tease you with Koray Kavukcuoglu,[565.032] [565.032][S01]director of research at DeepMind.[567.301] [567.301][S04] I remember training agents in the early days -[571.104] [571.104][S04]the first time actually those agents started behaving like okay,[576.777] [576.777][S04]it’s an environment, it’s trying to navigate,[578.946] [578.946][S04]it’s trying to avoid certain obstacles and what not -[581.715] [581.715][S04]the first time it starts doing that it’s actually-[584.852] [584.852][S04]it is nice, it’s like it’s quite fun to see that[587.821] [587.821][S04]because you know that it makes a decision for itself.[591.925] [591.925][S04]It think knowing that you have created an algorithm that can take decisions[597.464] [597.464][S04]I think that aspect is quite enjoyable.[599.132] [599.132][S04]That is very satisfactory. 10:00[600.534] [600.534][S01] It’s worth remembering that these games aren’t just a trivial pursuit for[604.304] [604.304][S01]DeepMind, they’ve invested in this rigorous training for a reason.[608.842] [608.842][S01]They want to see how an AI develops these kinds of skills for itself.[614.648] [614.648][S03] We spent a lot of time in this[616.383] [616.383][S03]Capture The Flag work looking into the the neural networks of these agents[620.521] [620.521][S03]to try and understand what they care about[621.989] [621.989][S03]and how they represent the game world.[624.157] [624.157][S03]And what was really cool is that we found that the agents[626.026] [626.026][S03]actually have a really, really rich representation of this game world[629.296] [629.296][S03]without being told anything about the game world itself.[631.231] [631.231][S03]You know it’s just look at the pixels of the screen[634.168] [634.168][S03]yet somehow they have clustered[635.569] [635.569][S03]the you know internal activations into things like oh[639.94] [639.94][S03]I’m in my home base, I’m in my opponent base,[642.943] [642.943][S03]I’ve got the flag and I can see my teammates ahead of me.[646.58] [646.58][S03]I’m looking at the opponent flag carrier while my teammates are holding my flag.[650.584] [650.584][S03]And you can even find individual neurons[652.319] [652.319][S03]which just activate if for example your teammates are holding the flag.[655.956] [655.956][S01] You can totally understand[657.824] [657.824][S01]how the agent is seeing the game as you go through.[661.328] [661.328][S03] I’m not sure about totally understand[663.063] [663.063][S03]but we’re really getting an idea of what is being represented strongly[667.1] [667.1][S03]and what isn’t being represented strongly.[669.203] [672.539][S01] Max’s agents are using something called neural networks.[676.009] [676.009][S01]It’s a type of machine learning algorithm that is loosely based[680.08] [680.08][S01]on a simplified version of the human brain.[683.383] [683.383][S01]Layers on layers of artificial neurons are connected together in a vast network[689.523] [689.523][S01]and fire information between themselves.[692.593] [692.593][S01]By looking inside an agent’s electronic brain,[696.763] [696.763][S01]Max can work out which micro level connections are responsible[701.068] [701.068][S01]for what macro level behavior. And this can be hugely beneficial[706.306] [706.306][S01]as AI becomes more integrated in our everyday lives.[710.878] [710.878][S03] The hope is that well into the future[713.78] [713.78][S03]we can start actually having agents which can go out into the real world[716.65] [716.65][S03]that can interact with humans, with other agents -[720.921] [720.921][S01] Without fighting -[722.022] [722.022][S03] Without fighting - being sensible, yeah.[724.992] [724.992][S03][Hannah chuckles] Not squabbling too much.[726.493] [726.493][S01] Unlike humans. [S03] yeah, exactly.[729.53] [731.899][S01] Games without frontiers team work without tears.[736.336] [736.336][S01]But there is a big leap between board games[738.438] [738.438][S01]or simple games like Capture The Flag[741.108] [741.108][S01]and the big bad world with all of its complexity and messiness.[746.446] [747.781][S01]You’ll remember David Silver - the man who brought us AlphaGo -[751.618] [751.618][S01]the agent that defeated the world champion[754.288] [754.288][S01]at the ancient board game of Go.[756.857] [756.857][S01]While he’s also involved in pushing DeepMind’s AI[760.727] [760.727][S01]into ever-more perplexing environments.[763.363] [763.363][S06] In the context of games, I think there is a further challenge[766.366] [766.366][S06]which is many people in the community are moving towards,[769.67] [769.67][S06]which is to take the most challenging computer game -[773.44] [773.44][S06]in this case it’s the game of StarCraft - and many people in the AI community[777.678] [777.678][S06]are viewing this as the next Grand Challenge -[780.013] [780.013][S06]now how can we actually devise agents[782.349] [782.349][S06]which can play in this very rich environment[785.485] [785.485][S06]which has challenges which are not only different[788.722] [788.722][S06]but many times vaster than Go in other ways.[791.992] [793.66][S01] This is DeepMind the podcast - an introduction to AI -[797.931] [797.931][S01]one of the most fascinating fields in science today.[801.134] [803.837][S01]Have you ever seen footage of those vast e-tournaments[807.174] [807.174][S01]where an entire arena of dedicated fans[809.843] [809.843][S01]excitedly watches on in support of highly skilled players,[814.615] [814.615][S01]sat on stage in their gaming chairs.[817.317] [817.317][S01]Armed only with a keyboard, a mouse and a computer screen.[821.355] [821.355][S01]Well chances are, they are playing something like[824.224] [824.224][S01]StarCraft II, created by the American video game[827.728] [827.728][S01]developer Blizzard entertainment.[829.496] [829.496][S01]It is a monumentally tricky tactical game[837.204] [837.204][S01]where you play as one of three races -[840.174] [840.174][S01]the enigmatically named Zerg, Protoss or Terrins.[845.212] [846.346][S01]Each player has to mine resources, build an economy[849.616] [849.616][S01]and acquire increasingly sophisticated technology,[852.92] [852.92][S01]all the time trying to defeat your alien opponents in a futuristic[858.125] [858.125][S01]rather bleak looking landscape.[860.294] [862.663][S01]Your field of view of the simulated game is limited by a moving camera[866.8] [866.8][S01]that you have to operate[868.302] [868.302][S01]and so there’s no way to see everything at once,[871.138] [871.138][S01]often you can’t see your opponent at all.[873.874] [873.874][S01]And it is played by 10s of thousands of people -[877.511] [877.511][S01]sometimes for hefty cash prizes.[880.247] [881.415][S01]And the human players are staggeringly fast.[884.751] [884.751][S01]The best in the world can manage up to 800 clicks in a minute.[890.09] [890.09][S01]Feeling inadequate?[891.692] [891.692][S10] Definitely super cool that I can work on one thing[895.362] [895.362][S10]that has been certainly a passion of mine in my teenage days.[900.0] [900.0][S01] Meet Oriol Vinyals, a research scientist at DeepMind.[903.904] [903.904][S01]He is an ex-pro StarCraft player and co-leads[907.241] [907.241][S01]the StarCraft effort at DeepMind.[909.61] [909.61][S10] as you develop a new algorithm, or a new idea, when you test it,[913.247] [913.247][S10]you actually see it play better the game you like,[916.216] [916.216][S10]so that’s very rewarding and very visual right,[918.519] [918.519][S10]that you try something new and you really see[921.221] [921.221][S10]oh my god, it’s really understands how this unit works.[925.626] [925.626][S01] StarCraft is a serious business - so serious in fact[928.829] [928.829][S01]that it has now been professionalized,[931.532] [931.532][S01]and for Oriol that proves that is a game that pushes human intelligence.[936.003] [936.003][S10] Humans found it interesting, so that means it’s an interesting game[940.24] [940.24][S10]that challenges intelligence and creativity[942.476] [942.476][S10]in ways that we like that we spend many hours playing.[945.579] [945.579][S01] So how good is the AI at the moment then?[948.148] [948.148][S01]How well can it play StarCraft?[950.017] [950.017][S10] It’s better than any AI anyone has ever built[953.487] [953.487][S10]and it obviously has learned from experience[956.39] [956.39][S10]not from someone knowing the game and encoding some set of rules.[960.494] [960.494][S10]This is I mean one of the most complicated games we’ve ever tackled -[964.498] [964.498][S10]it’s challenging kind of our understanding[967.367] [967.367][S10]and our algorithms quite a bit.[969.636] [969.636][S01] The DeepMind team started to see how good their work in progress really[974.141] [974.141][S01]was by inviting two of the world’s best[977.044] [977.044][S01]StarCraft II players to take on their own algorithm.[980.948] [981.982][S01]So let me introduce DeepMind’s AlphaStar -[984.885] [984.885][S01]the first artificial intelligence to ever take on top professional players.[990.524] [990.524][S01]It plays the full game of StarCraft II[993.126] [993.126][S01]by using a deep neural network trained directly from raw game data,[998.665] [998.665][S01]by supervised learning and reinforcement learning.[1001.568] [1002.336][S01]Your commentators are Dan Stemkoski - aka Artosis,[1007.407] [1007.407][S01]and Kevin van der Kooi - aka Rotterdam.[1011.645] [1011.645][S09] well first of all it’s really awesome to be here[1013.413] [1013.413][S09]together with you Dan, we’re both I think incredibly excited[1016.416] [1016.416][S09]to see how this evening unfolds.[1018.352] [1018.352][S07] I mean this is just so exciting that[1020.888] [1020.888][S07]DeepMind is doing all this -[1022.489] [1022.489][S01] Taking on AlphaStar in this benchmarking match is German champion[1026.66] [1026.66][S01]Dario Wünsch, better known as TLO. He’s normally a Zerg player,[1032.266] [1032.266][S01]but he’s playing as Protoss for this match.[1034.601] [1034.601][S01]Kevin and Dan are excited. Maybe even a tad over excited.[1039.706] [1039.706][S09] I’m so incredibly excited.[1041.308] [1041.308][S07] Oh my god - this is like the most exciting[1044.111] [1044.111][S07]I have personally ever been for an event.[1045.679] [1045.679][S09] I can’t wait to break down some -[1047.915] [1048.849][S07] So this is AlphaStar -[1050.784] [1050.784][S07]this is an AI that we don’t know how good it is yet,[1054.655] [1054.655][S07]but already we have some interesting things happening.[1058.225] [1058.225][S01] Now I’m not entirely conversant with the StarCraft playbook lingo here,[1063.43] [1063.43][S01]so I’ll just say that AlphaStar’s stalkers[1066.834] [1066.834][S01]are laying down some sharp moves.[1068.735] [1068.735][S07] it feels to me like so far these attacks have been very well[1072.406] [1072.406][S07]planned by AlphaStar.[1074.374] [1074.374][S09] and ah relentless - the last two attacks -[1076.677] [1076.677][S01] And in a matter of minutes, it’s all over.[1079.279] [1079.279][S07] Well that is it! [laughs][1081.615] [1081.615][S07]the GG is called the Good Game here from TLO[1085.519] [1085.519][S07]and the first game from AlphaStar against a pro gamer goes to AlphaStar.[1090.924] [1090.924][S01] David Silver was there, ringside.[1093.327] [1093.327][S06] We have a team that has been working on this[1094.962] [1094.962][S06]and ramping up our development over the last few months and this represents[1097.831] [1097.831][S06]ah you know a milestone where we actually for the first time[1100.701] [1100.701][S06]we saw an AI that was actually able to defeat a professional player.[1104.705] [1104.705][S01] Shall we have a quick word with our defeated challenger - TLO?[1107.908] [1107.908][S08] When I was practicing most of the humans[1109.71] [1109.71][S08]I played against played very standard StarCraft.[1112.246] [1112.246][S08]I once again I assumed after the first match[1115.182] [1115.182][S08]I probably have a good idea how to play against this agent, I did not.[1119.486] [1121.555][S01] Next up, the main event. AlphaStar versus Poland’s finest-[1126.46] [1126.46][S01]Grzegorz Komincz - better known as MaNa -[1129.63] [1129.63][S01]one of the world’s strongest professional[1131.832] [1131.832][S01]StarCraft players.[1133.934] [1133.934][S07] MaNa - I need to hear what you’re thinking here[1136.303] [1136.303][S07]- cause that looks scary.[1138.639] [1138.639][S05] Yeah, AlphaStar like he’s not scared about[1142.543] [1142.543][S05]ah the ramp, so if I would be playing against a human player[1146.58] [1146.58][S05]right there, nobody’s going up that ramp.[1149.316] [1149.316][S01] I should point for those of you who play[1151.051] [1151.051][S01]StarCraft that these matches are taking place under professional[1154.321] [1154.321][S01]match conditions on a competitive ladder map[1157.024] [1157.024][S01]and without any game restrictions. This version of AlphaStar[1160.494] [1160.494][S01]could see the whole of the game map at any one time,[1163.83] [1163.83][S01]but otherwise played in a comparable way to humans.[1167.267] [1167.267][S06] Our goal is not just to defeat these players.[1171.071] [1171.071]Our goal is to do it in the right way. [S10] two seconds guys![1175.142] [1175.142][S01] And the result - AlphaStar: 5; [S05] nil.[1179.947] [1182.649][S01] I should tell you that MaNa played a later version of the algorithm[1185.085] [1185.085][S01]in the end and won, so all in all, 5 - 1.[1189.523] [1191.592][S01]Now to understand how an AI could learn to play[1194.328] [1194.328][S01]StarCraft, Oriol Vinyals put me to the test.[1198.432] [1198.432][S01]A match to the end - mathematician versus machine.[1202.369] [1202.369][S01]I’ve got a quite a funky looking mouse in front of me and a normal keyboard,[1208.075] [1208.075][S01]and on the screen there is a very mean looking alien…[1212.479] [1212.479][S10] Yeah, Protoss.[1213.58] [1213.58][S01] I mean sort of sort of like an elephant meets -[1216.35] [1216.35][S01]um well he’s got fists.[1217.918] [1217.918][S01]I wouldn’t want to meet him on a dark night.[1219.186] [1219.186][S10] No. [S01] Is he my friend or not?[1221.154] [1221.154][S10] He is you. You’re going to be the commander of this particular race -[1225.859] [1225.859][S01] I quickly found out there is a lot to take in.[1229.73] [1229.73][S01]StarCraft is perhaps not for beginners.[1233.1] [1233.1][S01]You have your worker bees collecting resources for you.[1236.003] [1236.003][S01]Are these - are these all -[1237.271] [1237.271][S01]they’re almost like ant creatures running out and and grabbing crystals.[1241.475] [1241.475][S10] Right, exactly. [S01] And you need to try and work out[1243.21] [1243.21][S01]how your actions will affect your game in the future.[1245.779] [1245.779][S01]This is not easy for humans to learn let alone agents[1248.682] [1248.682][S01]who have absolutely no context, no object recognition,[1252.686] [1252.686][S01]and definitely no former StarCraft champion to hold their hand.[1257.224] [1257.224][S10] Hah! Look this is the enemy. [S01] oh no![1259.026] [1259.026][S10] Um it’s going to be pleasant - it just came to kind of find you[1262.863] [1262.863][S10]and now see what you’re doing which is absolutely nothing so far[1266.733] [1266.733][S10]so far we have done nothing at all -[1268.335] [1268.335][S01] Part of the challenge of StarCraft[1269.736] [1269.736][S01]is that there isn’t an idea strategy that wins every time.[1272.94] [1272.94][S01]it’s a bit like rock, paper, scissors in that way.[1275.209] [1275.209][S01]The winning tactic will depend on how your opponent plays.[1279.246] [1279.246][S01]But remember you only have a very narrow field of vision[1282.549] [1282.549][S01]- outside of where your camera is pointing,[1284.985] [1284.985][S01]your opponent could be up to anything.[1287.054] [1287.054][S10] Because you don’t see the other player, you must decide[1290.691] [1290.691][S10]when am I going to see it - do I already know what’s going on[1293.994] [1293.994][S10]and should I not go and scout what it’s doing[1296.997] [1296.997][S10]but maybe if I do that,[1298.398] [1298.398][S10]he knows that I know and so on and so forth,[1300.334] [1300.334][S10]so these kind of imperfect information aspect of StarCraft[1304.238] [1304.238][S10]is extremely interesting as a player and it’s going to be testing our agents[1309.243] [1309.243][S10]to levels that we haven’t seen in any other game.[1311.812] [1311.812][S10]And then of course there’s sort of details that have happening in the game[1315.215] [1315.215][S10]that you must remember for a long time.[1317.384] [1317.384][S01] Advice I should have listened to more carefully perhaps -[1320.654] [1320.654][S10] Ah - we’re being attacked and we’re probably going to die.[1322.456] [1322.456][S01] Oh no! Oh no! [laughs][1323.357] [1323.357][S10] Um, so that’s okay.[1324.892] [1324.892][S01] I didn’t last very long.[1326.126] [1326.126][S10] So but I think that this, this discovery phase[1328.095] [1328.095][S10]right where you would now basically you would lose,[1330.764] [1330.764][S10]you get the reward of minus one, and you start again.[1334.334] [1334.334][S01] If I was an algorithm, I wouldn’t be upset by losing,[1337.504] [1337.504][S01]I would just reset and go again.[1339.473] [1339.473][S01]Each time armed with a little more knowledge.[1342.709] [1342.709][S01]But to even be able to play[1344.178] [1344.178][S01]StarCraft in the first place, to even be able to operate the controls,[1348.615] [1348.615][S01]the AI had to master quite a few transferrable skills.[1352.586] [1352.586][S10] You’ve noticed when you were playing[1354.388] [1354.388][S10]that there were some movements that were resembling what it was like[1358.392] [1358.392][S10]to maybe navigate the web or like operate[1361.628] [1361.628][S10]um your laptop, namely click, drag and click,[1365.933] [1365.933][S10]drag and drop, like select rectangles and moving the mouse,[1370.404] [1370.404][S10]and maybe combining mouse with keyboard and so on,[1374.174] [1374.174][S10]and we tried exactly the same agent,[1376.343] [1376.343][S10]the same architecture absolutely everything the same the way,[1379.479] [1379.479][S10]the same code almost, and we changed, we changed the environment[1382.916] [1382.916][S10]instead of saying now here is the[1384.785] [1384.785][S10]StarCraft, please play to win, we said here is paint -[1390.157] [1390.157][S10]Microsoft Paint as an environment - interact with it[1393.293] [1393.293][S10]and I’ll reward you if what you paint looks like a face,[1396.73] [1396.73][S10]and it actually worked[1398.265] [1398.265][S10]so I think that’s just learning this basic skills of point[1402.436] [1402.436][S10]and click interfaces that applies so in so many places.[1405.739] [1405.739][S01] the same agent that plays[1407.674] [1407.674][S01]StarCraft can draw real faces in Microsoft Paint.[1411.812] [1411.812][S10] Right and here that the point to be clarified[1415.315] [1415.315][S10]is not the same agent that was trained to play[1418.185] [1418.185][S10]StarCraft, it’s the same algorithm that can train to play[1420.787] [1420.787][S10]StarCraft, can also train to do Paint.[1424.057] [1425.526][S01] Put that same algorithm to work drawing celebrities in Paint,[1429.796] [1429.796][S01]and it can capture all the main traits of the face.[1432.499] [1432.499][S01]Clicking and dragging the mouse to recreate shape[1435.569] [1435.569][S01]and tone and hairstyle, much like a street artist would.[1439.173] [1439.173][S10] It’s the same technique, but if you will,[1442.075] [1442.075][S10]it’s kind of a brain that is blank[1444.011] [1444.011][S10]and this brain can learn to do this and that and that[1446.313] [1446.313][S10]and then we kind of by acting in the environment repeatedly[1450.717] [1450.717][S10]and getting reward, the brain waits or gets shaped to do this task[1455.522] [1455.522][S10]or that task or that task -[1457.191] [1457.191][S10]we are not yet at the point where the same brain does both like we do[1461.695] [1461.695][S10]but obviously that’s one of the things[1463.363] [1463.363][S10]that we would be very interested in tackling next as well.[1466.767] [1466.767][S01] Because that’s stepping towards artificial general intelligence I guess.[1469.937] [1469.937][S10] Exactly. And that’s what we do every day.[1472.606] [1476.343][S01] That is the ultimate goal and it’s a topic of conversation[1479.713] [1479.713][S01]that’s never far away whoever in this building[1482.416] [1482.416][S01]you find yourself talking to[1484.384] [1484.384][S01]because the point of getting AI to play games like StarCraft[1487.521] [1487.521][S01]or Go is to enhance our understanding of what intelligence actually is.[1493.46] [1493.46][S01]Here’s Raia Hadsell from the Deep Learning team[1496.163] [1496.163][S02] we write programs, we run those programs,[1498.532] [1498.532][S02]those experiments where we might train an agent to play a game for instance[1503.504] [1503.504][S02]or to solve a puzzle in a simulated world[1506.907] [1506.907][S02]and then we look at the results of that.[1508.742] [1508.742][S02]It really is trying to understand this puzzle of learning and representation,[1513.847] [1513.847][S02]memory, control in terms of actions that a robot would take.[1517.851] [1517.851][S02]There’s so many complex parts to this big puzzle[1522.122] [1522.122][S02]of what is an intelligent being, what is an intelligent agent.[1526.193] [1527.294][S01] But if you ask people what they think the future of AI looks like,[1531.698] [1531.698][S01]it tends to be wrapped up in something a bit more physical,[1535.435] [1535.435][S01]something that comes complete with moving arms and everything.[1539.573] [1539.573][S06] I think one natural challenge for AI[1544.044] [1544.044][S06]which many people are centering upon[1547.014] [1547.014][S06]would be to actually have an impact on the real world[1551.251] [1551.251][S06]in the guise of robotics to actually see a robot[1554.354] [1554.354][S06]which is able to move, to grip, to manipulate,[1557.524] [1557.524][S06]to even have locomotion in anything approaching[1560.894] [1560.894][S06]not even what a human does - maybe even an animal.[1563.23] [1563.23][S06]I think this this would represent a major stride forwards.[1566.667] [1566.667][S01] More on that next time.[1568.635] [1570.504][S01]If you would like to find out more about the themes in this episode,[1574.408] [1574.408][S01]or explore the world of AI research beyond[1577.077] [1577.077][S01]DeepMind, you’ll find plenty of useful links[1579.646] [1579.646][S01]in the show notes for each episode - and if there are stories or resources[1583.784] [1583.784][S01]that you think other listeners would find helpful, then let us know.[1587.054] [1587.054][S01]You can message us on Twitter or email the team at podcast@deepmind.com.[1592.593] [1592.593][S01]You can also use the address to send us your questions[1595.429] [1595.429][S01]or feedback on the series.[1597.464] [1598.198][S01]But for now, let’s nip out for a bit of air.[1600.601]


[0.0][S02] Welcome back to "Google DeepMind--[2.13] [2.13][S02]The Podcast," with me, your host, Professor Hannah Fry.[5.745] [5.745][MUSIC PLAYING][9.705] [13.18][S02]I think it's fair to say that generative AI, as we have it[16.059] [16.059][S02]now, is a bit of a mismatch for our science fiction fantasies[20.5] [20.5][S02]about what technology might become.[23.18] [23.18][S02]But it is worth remembering that this isn't the only possibility.[26.33] [26.33][S02]There are other forms of AI forms,[28.61] [28.61][S02]forms which are by their very design, autonomous entities.[32.509] [32.509][S02]These AI are firmly in the realm of robot butlers[35.92] [35.92][S02]or a virtual concierge that anticipates your every whim[39.43] [39.43][S02]before you even have it.[41.51] [41.51][S02]And OK, we might have to wait a little while for those two[44.53] [44.53][S02]particular examples.[46.01] [46.01][S02]But I'm talking here about artificial intelligence[49.24] [49.24][S02]with agency, AI that has built-in wants and objectives[54.28] [54.28][S02]and the ability to independently make decision after decision[58.54] [58.54][S02]in pursuit of its own goals.[61.25] [61.25][S02]This kind of AI are known as agents,[64.42] [64.42][S02]and while agents haven't yet exploded[66.46] [66.46][S02]into the public consciousness quite as much[68.84] [68.84][S02]as generative AI has, trust us, they[71.84] [71.84][S02]are going to be a very big deal.[74.64] [74.64][S02]And in the meantime, a very good training ground for agents[77.72] [77.72][S02]is the world of video games.[79.74] [79.74][S02]Think about it-- perfectly packaged,[82.08] [82.08][S02]neatly constrained environments where the agents can run wild,[85.84] [85.84][S02]work out the rules for themselves,[87.44] [87.44][S02]and learn how to handle autonomy.[90.19] [90.19][S02]If you've been following us since the beginning,[92.19] [92.19][S02]then you will know that AI in games[94.1] [94.1][S02]is something that DeepMind has a proud heritage in.[97.44] [97.44][S02]And so to explain today, we are joined[100.25] [100.25][S02]by Frederic Besse, who is a Senior Staff Research[103.01] [103.01][S02]Engineer at Google DeepMind.[104.76] [104.76][S02]Frederic has a PhD in Computer Vision[106.94] [106.94][S02]and was one of the earliest people[108.59] [108.59][S02]to join the DeepMind team back in 2015,[111.18] [111.18][S02]when there were only about 100 researchers here.[114.08] [114.08][S02]And before the decade he spent at the cutting-edge[116.33] [116.33][S02]of artificial intelligence, he also[118.31] [118.31][S02]spent some time working in the visual effects[120.38] [120.38][S02]industry on how to integrate CGI into real footage.[124.71] [124.71][S02]Frederic, thank you so much for joining us.[126.95] [126.95][S02]With a CV like that, I can see why they wanted you.[129.83] [129.83][S01] Well, thanks.[131.038] [131.038][S02] OK, well, so I think it's probably[132.955] [132.955][S02]best if we start off with some definitions.[134.86] [134.86][S01] Sure.[135.21] [135.21][S02] So, I mean, I've talked briefly about[137.54] [137.54][S02]what agents are there, but how do you define them?[140.16] [140.16][S01] So, in my mind, an agent[142.4] [142.4][S01]is a very general concept.[144.33] [144.33][S01]And I define it as an entity that can act in an environment.[149.13] [149.13][S01]I think action is what defines an agent.[151.73] [151.73][S01]And an environment, on the other side,[153.41] [153.41][S01]is a place which can provide observations for the agent[159.05] [159.05][S01]and which the agent can interact with.[161.16] [161.16][S01]And so what's important to understand[165.59] [165.59][S01]is that the agent, by acting into the environment,[167.85] [167.85][S01]will change the environment, which[169.94] [169.94][S01]is crucial for developing AI systems that[173.33] [173.33][S01]can do useful things and understand[175.4] [175.4][S01]the consequences of their actions[177.32] [177.32][S01]in environments, in games, in the world,[180.65] [180.65][S01]on the internet, et cetera.[181.858] [181.858][S02] This sounds like something that[183.65] [183.65][S02]exists beyond just AI, though.[185.19] [185.19][S02]I mean, autopilot on planes, would that count as an agent?[187.757] [187.757][S01] Yes, absolutely.[189.09] [189.09][S01]So maybe we can give a few examples of agents[191.19] [191.19][S01]because it's a very general concept.[192.69] [192.69][S01]So humans are agents because we can act.[195.56] [195.56][S01]We observe the world with our eyes and our ears.[198.18] [198.18][S01]We have proprioception.[199.38] [199.38][S01]We know where our limbs are.[200.61] [200.61][S01]So we are agents.[201.39] [201.39][S01]We can affect the world.[202.83] [202.83][S01]Similarly, robots as well.[204.36] [204.36][S01]But if you go back to maybe decades ago[208.1] [208.1][S01]where we had agents such as Autopilot, which we handcrafted[212.78] [212.78][S01]the logic of those agents, we were[215.51] [215.51][S01]programming them to do what we wanted them to do.[218.46] [218.46][S01]So they were different from the modern agents[221.06] [221.06][S01]that we can talk about later on, but these were indeed[224.57] [224.57][S01]examples of agents.[225.567] [225.567][S02] So, all right.[226.65] [226.65][S02]If that's an agent then, is that the same thing[229.46] [229.46][S02]as autonomy or having agency?[232.04] [232.04][S02]Are these all slightly, subtly different?[233.94] [233.94][S01] So I think they are subtly different.[236.148] [236.148][S01]I think an agent can act and do things[239.51] [239.51][S01]that it has been programmed to do,[244.07] [244.07][S01]and that's agency, being able to take actions.[246.9] [246.9][S01]Now autonomy is more the skill or the property[251.71] [251.71][S01]of acting by itself to accomplish a certain task.[257.06] [257.06][S01]And there are various levels of autonomy.[259.089] [259.089][S01]So if we have a self-driving car, which will take decision[264.04] [264.04][S01]to best navigate the roads, you could[266.98] [266.98][S01]say that there is a certain level of autonomy.[269.8] [269.8][S01]Some agents are much less autonomous[272.41] [272.41][S01]and will require a lot more of your input.[274.94] [274.94][S01]And then on the other end of the spectrum,[276.703] [276.703][S01]you could have an agent that goes in the world[278.62] [278.62][S01]and learn new things, explore, which we don't really[283.63] [283.63][S01]have to instruct much.[286.19] [286.19][S01]So there are a spectrum in the space of autonomy.[289.053] [289.053][S02] Is part of that almost[290.47] [290.47][S02]a hierarchy of the type of objectives that you're setting?[293.7] [293.7][S02]Like, I don't know, the autonomous action[295.595] [295.595][S02]of picking up a cup and putting it down[297.22] [297.22][S02]is slightly different to deciding what's for lunch.[299.8] [299.8][S01] Yes, because we are training agents[302.68] [302.68][S01]to do simple things because this is a starting point.[306.72] [306.72][S01]But having an agent that can cook for you your whole lunch--[309.462] [309.462][S02] And decide what you want as well would be nice.[311.92] [311.92][S01] Yes, as well.[313.128] [313.128][S01]It would be a recommender system will require a lot of sub steps[315.9] [315.9][S01]and sub tasks too, and solve those tasks properly[318.66] [318.66][S01]as well first.[320.01] [320.01][S01]So we call those high-level tasks[323.19] [323.19][S01]compared to short-horizon tasks, which will be very basic,[327.33] [327.33][S01]almost mechanical actions, like grab an object,[330.85] [330.85][S01]put the objects on the table, compared to cook lunch.[335.105] [335.105][S02] Yeah, like the robots[336.48] [336.48][S02]you see in factories, for example, versus the type of ones[338.94] [338.94][S02]that we have in science fiction.[340.42] [340.42][S02]But is that the aim?[342.04] [342.04][S02]Are we talking about robots eventually[344.52] [344.52][S02]that will be in our homes?[345.805] [345.805][S01] I think so.[346.93] [346.93][S01]So what we are trying to do is to develop general agents.[350.22] [350.22][S01]And we want to reach a point, which[353.19] [353.19][S01]we call AGI, Artificial General Intelligence, which[356.34] [356.34][S01]is a point where we will have agents and AI systems that will[361.17] [361.17][S01]be truly general, as general as humans,[364.03] [364.03][S01]will be able to adapt to new situations, reason like we do,[367.57] [367.57][S01]and even surpass our capabilities in doing tasks.[372.17] [372.17][S01]Once we reach this level of competency in AI systems,[379.02] [379.02][S01]this will unlock a lot of potential[381.13] [381.13][S01]for what we can do as humans.[383.71] [383.71][S01]Those AI systems will be able to accelerate[387.22] [387.22][S01]our scientific progress, for example,[389.29] [389.29][S01]or do a lot of tasks which maybe we are not very good for.[393.49] [393.49][S01]For example, maybe driving, or help you in your home,[398.83] [398.83][S01]help you do shopping online.[400.63] [400.63][S01]I think there are some applications which we can't even[403.18] [403.18][S01]begin to conceive because this is such[405.52] [405.52][S01]a technological leap that I think we will just discover it[410.02] [410.02][S01]as we go.[410.9] [410.9][S01]But we do believe that building such general agents, which[414.85] [414.85][S01]can act into an environment, is key to achieve AGI.[418.662] [418.662][S02] That's interesting in the way[420.37] [420.37][S02]that you're describing it, that some of these agents[423.04] [423.04][S02]are going to be virtual, essentially,[424.778] [424.778][S02]and then some of them will be actually physically[426.82] [426.82][S02]embedded in an environment.[428.3] [428.3][S01] Yes.[429.133] [429.133][S01]So I think it all goes back to what's your observation space[434.26] [434.26][S01]and what's your action space.[435.94] [435.94][S01]So we can have physical agents, embodied physical agents,[440.08] [440.08][S01]like robots, which will be able to do things in the real world,[443.15] [443.15][S01]which will be very useful.[444.5] [444.5][S01]And we already, as you--[445.595] [445.595][S02] My robot butler, thank you.[447.22] [447.22][S01] Exactly.[448.9] [448.9][S01]But also virtual agents, which work not in the real world,[453.2] [453.2][S01]but in an abstract virtual space.[456.7] [456.7][S01]This could be still a physical space, for example, games,[459.77] [459.77][S01]which we use in our research to also demonstrate and understand[464.71] [464.71][S01]what those agents can do.[466.66] [466.66][S01]But also, you could have an agent that[468.52] [468.52][S01]doesn't have any physical, even virtual action space,[473.18] [473.18][S01]but instead it can make programs.[476.15] [476.15][S01]For example, write Python code, build software for you,[479.89] [479.89][S01]which is not a physical space, but it's still an agent[482.41] [482.41][S01]because it can take actions, it can compile the function,[485.93] [485.93][S01]it can look on the internet.[488.26] [488.26][S01]And so, this is still very valuable.[490.978] [490.978][S02] How is this different to the chatbots[493.02] [493.02][S02]that we've seen from large language models?[494.65] [494.65][S02]Because they can do a lot of that stuff too, right?[496.69] [496.69][S02]They can search the internet, they[498.107] [498.107][S02]can understand your instructions and make decisions, sort of?[501.647] [501.647][S01] Yes.[502.48] [502.48][S01]So chatbots, what you get out of chatbots generally[506.73] [506.73][S01]is a chat output.[509.11] [509.11][S01]So they produce natural language.[511.29] [511.29][S01]Now, there are some capabilities in some chatbots[515.37] [515.37][S01]which could go and search the internet for you.[519.0] [519.0][S01]But this space is still fairly constrained[523.559] [523.559][S01]to just being either natural language or specific functions[530.34] [530.34][S01]that the programmer adds to the chatbot to do things.[534.79] [534.79][S01]Now those chatbots, when they speak with you,[537.82] [537.82][S01]I mean, in a way, you are the environment of the chatbot,[540.25] [540.25][S01]right?[540.75] [540.75][S01]What the chatbot says will affect you[542.49] [542.49][S01]and will affect what you ask the chatbot next.[545.74] [545.74][S01]But maybe there is not as much consequence[550.06] [550.06][S01]as if the chatbot or the agent was[551.86] [551.86][S01]acting in an environment which drastically changes when[556.36] [556.36][S01]the agent takes some action.[557.93] [557.93][S01]So it's a bit more of a passive type of agent.[561.11] [561.11][S01]We could call it an agent, but it's[562.75] [562.75][S01]very different from an agent that acts in the real world.[566.96] [566.96][S01]And the way also those models are trained,[570.28] [570.28][S01]they are trained with a lot of data that[573.19] [573.19][S01]are scraped from the internet.[575.89] [575.89][S01]But they are not trained in an environment[578.14] [578.14][S01]to try to do things, fail, understand why they failed,[581.65] [581.65][S01]and try again.[582.41] [582.41][S01]So this is a difference between some[585.64] [585.64][S01]of the way those models are trained with human data.[588.77] [588.77][S02] So then, should we be thinking of these as separate[592.095] [592.095][S02]or can you combine the two?[593.22] [593.22][S02]Can you combine large language models and an agent[595.4] [595.4][S02]that has an objective?[596.4] [596.4][S01] So you can actually,[598.28] [598.28][S01]there are some robotics paper from Google DeepMind[601.73] [601.73][S01]that is called RT-2 and RT-X, which are doing that.[605.185] [605.185][S01]So they are training.[606.06] [606.06][S01]So a language model has a lot of knowledge[609.14] [609.14][S01]from the internet scale data that it has been trained on.[612.2] [612.2][S01]So if you want, it has a lot of common sense[614.81] [614.81][S01]and it understands a bunch of concepts,[616.47] [616.47][S01]but it doesn't have the ability to actuate joints of a robot.[620.13] [620.13][S01]But you can text this model and train[622.76] [622.76][S01]it to produce joint activation for your robotic arm,[625.65] [625.65][S01]for example.[626.15] [626.15][S01]That's what they've done in that paper.[628.64] [628.64][S02] I guess almost the language model gives[631.52] [631.52][S02]the algorithm some sort of inverted commas,[634.83] [634.83][S02]conceptual understanding of things.[636.63] [636.63][S01] Yes, I think that's what is great[639.17] [639.17][S01]about those big models is that they have such knowledge[642.72] [642.72][S01]and understand concepts and words[646.16] [646.16][S01]and what you chat to them about that they can make plans.[650.74] [650.74][S01]But all of this is in language space.[652.35] [652.35][S01]And we need to bridge the gap towards acting in the real world[658.95] [658.95][S01]or in the virtual world.[660.027] [660.027][S02] To go off and actually enact it?[661.86] [661.86][S01] Yes.[662.958] [662.958][S02] What is it about games, though?[664.75] [664.75][S02]Why is it useful to start with games?[666.97] [666.97][S01] So I think games offer a very rich experience[672.51] [672.51][S01]and games are very varied.[674.05] [674.05][S01]So if you look at how many games there[675.72] [675.72][S01]are on computers, for example, you have tens of thousands[679.74] [679.74][S01]of games.[680.41] [680.41][S01]And each game offers you different scenarios,[683.203] [683.203][S01]different environments, different things[684.87] [684.87][S01]you could be doing in the game.[686.185] [686.185][S01]So we think it's a great proving ground[687.81] [687.81][S01]to train AI systems to match human performance.[692.61] [692.61][S01]There are other advantages in games.[694.87] [694.87][S01]So they are virtual, so they are easy to scale.[697.21] [697.21][S01]You could instantiate a lot of different games in parallel[701.13] [701.13][S01]compared to physical systems, which are[703.5] [703.5][S01]constraints to the real world.[705.9] [705.9][S01]They are also safe.[707.55] [707.55][S01]So if things go wrong in games, it doesn't really matter.[710.53] [710.53][S01]It's just a game.[712.5] [712.5][S01]Games are a place where humans can[714.72] [714.72][S01]interact with agents, for example, in multiplayer games.[719.31] [719.31][S01]And games are fun as well for humans to play.[723.76] [723.76][S01]And I think it's a great source of experience data[726.6] [726.6][S01]that we can collect.[727.602] [727.602][S02] Because you've already got agents already[729.81] [729.81][S02]in non-player characters, NPCs floating around the place.[732.738] [732.738][S01] Yes, yes.[733.78] [733.78][S01]So I think that's a very interesting point.[739.03] [739.03][S01]Historically, and actually it's still the case,[741.75] [741.75][S01]when game developers create agents inside of games,[745.18] [745.18][S01]it's a very different beast to the type of agents that we make.[749.14] [749.14][S01]Those agents have access to the internal code of the game.[752.62] [752.62][S01]And the game developers just write the logic, which[756.6] [756.6][S01]works great, and they are fast and they can[758.85] [758.85][S01]do pretty impressive things.[761.218] [761.218][S02] Things like stand around,[762.76] [762.76][S02]and then if someone approaches you, attack them.[764.79] [764.79][S01] Yeah, there is some pretty complicated logic[768.42] [768.42][S01]you could put in the game, in the agents in the game,[770.77] [770.77][S01]in the NPCs.[771.55] [771.55][S01]But there is a limit in how complicated you can describe[775.68] [775.68][S01]a behavior in terms of code.[777.52] [777.52][S01]And that's where deep learning and modern agents come in,[781.21] [781.21][S01]because those are not-- the logic inside those agents[784.41] [784.41][S01]is not handcrafted by humans.[786.46] [786.46][S01]Instead, we subject the agent to a lot of experience data,[790.29] [790.29][S01]and the agent will learn the behavior for itself[793.98] [793.98][S01]in a softer way with more nuances.[797.41] [797.41][S01]And it can pick up on very complex behavior, which[800.64] [800.64][S01]maybe wouldn't be feasible to write down in code,[804.71] [804.71][S01]in programming language.[805.71] [805.71][S02] Rather than just following rules,[806.95] [806.95][S02]it's sort of working out how to achieve its objective itself?[809.492] [809.492][S01] Yeah.[810.45] [810.45][S02] DeepMind's got quite a long history[812.73] [812.73][S02]of playing around with games.[815.62] [815.62][S02]Maybe that a bit trivializes it slightly[817.47] [817.47][S02]considering the things that they've achieved by doing so.[819.9] [819.9][S01] Yes.[820.733] [820.733][S01]So back in 2015, there was the first breakthrough[826.05] [826.05][S01]in neural network agents, called DQN, for Deep Q Network,[831.15] [831.15][S01]in which the agent was trained to play Atari.[835.05] [835.05][S01]And the breakthrough here is that the agent was only[838.5] [838.5][S01]using pixels as the observation space.[841.6] [841.6][S02] So Atari is like "Space Invaders,"[844.17] [844.17][S02]break out the one with the bricks where you've got a paddle[847.245] [847.245][S02]and a ball bouncing around.[848.37] [848.37][S02]Like, the games you had in the 1980s.[850.78] [850.78][S01] Yeah, so the arcade games, exactly.[853.987] [853.987][S02] And the only thing it would take as an input[856.32] [856.32][S02]were the pixels on the screen?[857.98] [857.98][S01] So the pixels of the screen.[859.06] [859.06][S01]And there was also the concept of reward, which is[861.48] [861.48][S01]like how many points you get.[862.87] [862.87][S01]And that is very important for those agents to learn.[865.5] [865.5][S01]Because what those agents are doing and how they are trained,[868.93] [868.93][S01]they are trained using a method called reinforcement learning,[871.9] [871.9][S01]which aims at maximizing this score, maximizing the reward.[876.85] [876.85][S01]And so combining deep learning with deep neural network[880.14] [880.14][S01]and reinforcement learning yielded the DQN agent,[883.51] [883.51][S01]which was first of its kind.[885.49] [885.49][S01]And that was a very big milestone for DeepMind.[890.27] [890.27][S01]Then another important agent that was built[895.34] [895.34][S01]was AlphaGo, which was an agent that[898.79] [898.79][S01]tries to learn to play the game of Go, which is a board game.[903.89] [903.89][S02] Notoriously difficult.[905.39] [905.39][S01] Notoriously difficult.[906.973] [906.973][S01]I think it was uncertain whether AI could ever[910.85] [910.85][S01]beat the game of Go.[912.5] [912.5][S01]But then that was proven to be wrong.[916.58] [916.58][S02] When AlphaGo beat Lee Sedol.[918.36] [918.36][S01] Yes, exactly.[919.568] [919.568][S01]So that was a huge result. Then if you continue[922.645] [922.645][S01]along the history, so this is a non-exhaustive list[924.77] [924.77][S01]because there is a lot of research that[925.91] [925.91][S01]has been happening also in between all[927.95] [927.95][S01]these important milestones.[930.32] [930.32][S01]There was AlphaZero, which is taking the agents that play Go,[935.9] [935.9][S01]and also it could play chess and Shogi, to the next level.[939.39] [939.39][S01]And this agent was trained without any human data.[942.12] [942.12][S01]So while AlphaGo was trained using[943.7] [943.7][S01]a mix of human data and self-play and reinforcement[946.13] [946.13][S01]learning, AlphaZero was trained without any human data.[948.98] [948.98][S01]And just starting from scratch with self-play,[951.31] [951.31][S01]it discovered the best chess moves,[954.36] [954.36][S01]and that was very interesting to see.[957.523] [957.523][S02] Extraordinary, that taking the human input out[959.94] [959.94][S02]made it better.[961.44] [961.44][S01] Well, yes.[963.87] [963.87][S01]So it's interesting because you can actually[966.15] [966.15][S01]look at what are the most optimal moves for chess,[969.81] [969.81][S01]which is a fascinating find.[972.898] [972.898][S02] But then it moved into computer games[974.94] [974.94][S02]again, didn't it?[976.027] [976.027][S01] Yes.[976.86] [976.86][S01]So I think one next very important paper was[981.09] [981.09][S01]the AlphaStar paper, which is an agent that is playing "StarCraft[986.91] [986.91][S01]II," which is a game, competitive game.[989.91] [989.91][S01]That's very different environment[991.44] [991.44][S01]from, say, Go or Chess.[993.49] [993.49][S01]So one of the challenge here is that in "StarCraft II,"[997.33] [997.33][S01]you only see partial information from your environment.[1000.215] [1000.215][S01]There is this thing called fog of war where you cannot see what[1002.84] [1002.84][S01]your enemy is doing.[1004.23] [1004.23][S01]And so you really have to take this into account[1007.01] [1007.01][S01]during your strategy, while in Go, in chess, you[1012.38] [1012.38][S01]see the whole board in front of you.[1013.88] [1013.88][S01]So you have access to the whole state of the game.[1016.07] [1016.07][S01]Also, the agent has to play in real time because "StarCraft[1019.454] [1019.454][S01]II," once you play against a human at test time,[1021.98] [1021.98][S01]you really have to play as fast as you can,[1024.19] [1024.19][S01]so that you cannot take minutes to think.[1027.13] [1027.13][S01]And the "StarCraft II" agent was able to beat very,[1030.079] [1030.079][S01]very good humans at the game.[1032.14] [1032.14][S02] So now this is one step on even further.[1034.46] [1034.46][S02]So it's like at every step, you remove[1037.06] [1037.06][S02]something that made it easy.[1038.42] [1038.42][S02]So Atari, you get points for every action.[1040.579] [1040.579][S02]chess, you don't.[1041.66] [1041.66][S02]In chess, you get to see the entire board.[1045.17] [1045.17][S02]In "StarCraft," you don't.[1046.54] [1046.54][S02]In "StarCraft," it's competitive,[1048.46] [1048.46][S02]in what you're doing now, it isn't.[1050.537] [1050.537][S01] Yes.[1051.37] [1051.37][S01]So those agents were trying to maximize the score[1054.97] [1054.97][S01]and to win the game, while what we do in SIMA is very different.[1058.88] [1058.88][S01]We train agents to follow instructions.[1061.16] [1061.16][S02] What's this project called?[1062.83] [1062.83][S01] Our project is called SIMA.[1064.622] [1064.622][S01]That stands for Scalable Instructable Multi-world Agent.[1068.83] [1068.83][S01]And we used various different video games and also research[1075.31] [1075.31][S01]environments to train an agent to follow instructions[1079.69] [1079.69][S01]that we ask it to do.[1081.26] [1081.26][S01]Those instructions are fairly simple that you can typically[1084.34] [1084.34][S01]carry out in 5 to 10 seconds, such as pick up the apple,[1087.97] [1087.97][S01]or turn around, climb up the ladder, et cetera.[1091.46] [1091.46][S01]And so there is no score.[1093.19] [1093.19][S01]And we use the games really as containers,[1095.21] [1095.21][S01]as walls in which we can carry out[1097.42] [1097.42][S01]a whole bunch of instructions.[1098.885] [1098.885][S01]And the amount of instruction, the number of instructions[1101.26] [1101.26][S01]that you can come up with is very large due to the fact[1105.52] [1105.52][S01]that we use natural language to formulate those instructions.[1109.84] [1109.84][S01]So this poses other challenges.[1113.89] [1113.89][S01]We really want to make our agents to be general.[1116.54] [1116.54][S01]So while a Go agent only learns to beat the game of Go,[1121.48] [1121.48][S01]we are interested in making an agent that can really[1124.63] [1124.63][S01]understand what you're saying, understand the context,[1127.28] [1127.28][S01]understand the environment, and then carry out this instruction.[1130.06] [1130.06][S01]And so we are aiming for generality.[1132.0] [1132.0][S01]And I think generality is a key property that we want[1135.98] [1135.98][S01]in our agents to pursue AGI.[1138.693] [1138.693][S02] What do you mean by sandbox games?[1140.61] [1140.61][S01] So a sandbox game[1141.985] [1141.985][S01]is a game where the player can basically[1144.38] [1144.38][S01]do whatever they want within the space of what the game offers.[1148.53] [1148.53][S01]So it's a game which has natural landscape.[1151.62] [1151.62][S01]You can craft, you can build, you can cook, you can explore.[1155.45] [1155.45][S02] Like "Minecraft."[1156.685] [1156.685][S01] Like "Minecraft."[1158.06] [1158.06][S01]That's what we call a sandbox game because there is not really[1160.7] [1160.7][S01]a goal to the game.[1162.81] [1162.81][S01]Some games are sandbox with a goal, so it's a bit of a mix.[1166.82] [1166.82][S01]But yeah, you just have fun and do a bunch of things.[1170.077] [1170.077][S02] Like in an actual sandbox.[1171.66] [1171.66][S01] Exactly, yes.[1172.76] [1172.76][S02] Well, how about you?[1173.64] [1173.64][S02]Are you a gamer?[1174.6] [1174.6][S01] Yes, very much so.[1176.73] [1176.73][S01]I was given a Gameboy for my sixth birthday.[1178.805] [1178.805][S02] Sixth birthday?[1179.93] [1179.93][S01] Yes.[1180.92] [1180.92][S02] Lucky thing.[1181.92] [1181.92][LAUGHTER][1182.72] [1182.72][S01] Yeah, it was amazing.[1184.262] [1184.262][S01]I think my first game was "Tetris."[1186.11] [1186.11][S01]And yeah, I was amazed by being able to play[1191.0] [1191.0][S01]those games in the car.[1192.65] [1192.65][S01]Car trips were never the same.[1194.74] [1194.74][S02] Yeah, that's true.[1195.99] [1195.99][S02]And as an adult, did you go beyond "Tetris?"[1197.908] [1197.908][S01] Yes, of course.[1199.2] [1199.2][S01]I remember when we at home got DSL,[1203.27] [1203.27][S01]which could give you basically constant internet access.[1206.463] [1206.463][S01]I was very happy because I could finally[1208.13] [1208.13][S01]play online games without being worried[1210.14] [1210.14][S01]about the counter, how much time do I spend on the internet.[1215.51] [1215.51][S01]So yeah, online games, competitive games,[1218.22] [1218.22][S01]sandbox games, and [INAUDIBLE] games, all of this I[1221.93] [1221.93][S01]do quite like.[1223.02] [1223.02][S02] I mean, so you've got[1224.395] [1224.395][S02]basically the perfect experience then to be in this space.[1226.98] [1226.98][S01] Looks like it.[1228.863] [1228.863][S02] The new projects that you're[1230.53] [1230.53][S02]working on now, these sandbox games,[1232.34] [1232.34][S02]I mean, they are quite a big departure.[1234.47] [1234.47][S02]How hard is it to get an agent to achieve things[1237.497] [1237.497][S02]in an environment where you don't have[1239.08] [1239.08][S02]that really clearly defined metric of success?[1241.14] [1241.14][S01] Yes, so it is a very different style of building[1245.71] [1245.71][S01]an agent training from before.[1247.39] [1247.39][S01]There is some prior work, which we were very inspired by,[1253.19] [1253.19][S01]which was the interactive agents project,[1255.01] [1255.01][S01]there is a blog post on the DeepMind blog post, where[1259.0] [1259.0][S01]they use the Playhouse environment[1261.43] [1261.43][S01]with a bunch of objects, and the goal[1263.29] [1263.29][S01]was to get the agent to carry out instructions in that house.[1267.173] [1267.173][S02] It's like a simulated environment?[1269.09] [1269.09][S01] It's a simulated environment, yes.[1270.25] [1270.25][S01]It's like a children's house with colorful objects.[1272.99] [1272.99][S01]And the goal was to take the teapots and put it on the beds,[1277.07] [1277.07][S01]and there was no reward as well.[1279.86] [1279.86][S01]So we wanted to take this to the next level.[1282.89] [1282.89][S01]And we also wanted to use games that are not games[1286.93] [1286.93][S01]that we make ourselves, that are available for everyone to play.[1290.235] [1290.235][S02] So basically it's the games[1291.86] [1291.86][S02]that you guys are playing?[1293.678] [1293.678][S02]Is that what it is?[1294.47] [1294.47][S01] Some of them are, indeed.[1296.178] [1296.178][LAUGHTER][1296.888] [1296.888][S02] I just want more time to spend on this game[1299.18] [1299.18][S02]that I've been doing, so I'm going[1300.597] [1300.597][S02]to build an agent to do it.[1301.795] [1301.795][S02]That's essentially what it sounds like.[1303.23] [1303.23][S01] Yeah, you could do it.[1304.813] [1304.813][S01]And so, yeah, we got into partnership[1309.5] [1309.5][S01]with game developers and game studios[1312.47] [1312.47][S01]to be able to use their games in our research.[1314.76] [1314.76][S01]And we picked games which are of a certain genre.[1319.52] [1319.52][S01]So we wanted 3D games because we were interested in these[1323.81] [1323.81][S01]embodied 3D setup.[1326.495] [1326.495][S01]We wanted to be able to define our own objective with language.[1329.12] [1329.12][S01]So if you go into "Valheim," for example,[1331.71] [1331.71][S01]which is a Viking survival simulation,[1334.01] [1334.01][S01]you can make up a lot of tasks for the agents.[1336.03] [1336.03][S01]You can say, go get some mud, build a house,[1339.51] [1339.51][S01]craft jackets, et cetera.[1342.32] [1342.32][S02] Then how on Earth do you train an agent[1344.69] [1344.69][S02]when you don't have rewards?[1345.99] [1345.99][S01] So what we did is that we collected data[1348.35] [1348.35][S01]from humans playing the game.[1350.27] [1350.27][S01]And then we used a technique called imitation learning, which[1354.62] [1354.62][S01]teaches the agent to just mimic and predict[1357.56] [1357.56][S01]what a human would do next.[1359.16] [1359.16][S01]So given all of the observation, given what you've done before,[1362.25] [1362.25][S01]what would the human do?[1364.4] [1364.4][S01]And that's how we train the behavior of the agent.[1367.44] [1367.44][S01]So there is no reward.[1369.26] [1369.26][S01]We just imitate humans, which-- yeah.[1371.21] [1371.21][S02] Is it probabilistic, then?[1372.793] [1372.793][S02]Is it sort of like, in this sort of situation,[1375.05] [1375.05][S02]humans acted in this way, so we'll[1376.987] [1376.987][S02]act in that way with this probability?[1378.57] [1378.57][S01] Yes, exactly.[1379.11] [1379.11][S01]So all the actions of the agent are probabilistic.[1381.51] [1381.51][S01]So before the agent acts, there is a probability distribution,[1386.19] [1386.19][S01]which needs to sample and pick an action from.[1388.86] [1388.86][S01]Often it's the most likely action,[1390.387] [1390.387][S01]but sometimes you can take an unlikely action, which can[1392.72] [1392.72][S01]lead to unforeseen behavior.[1396.15] [1396.15][S01]Sometimes it's a good behavior.[1397.79] [1397.79][S01]Sometimes it's not.[1399.765] [1399.765][S02] Go on, give you an example.[1401.39] [1401.39][S01] Well, so for example,[1402.932] [1402.932][S01]we have a game such as "Goat Simulator."[1405.9] [1405.9][S02] Tell us the objective of "Goat Simulator."[1408.15] [1408.15][S01] So the objective of "Goat Simulator"[1410.317] [1410.317][S01]is to basically cause mayhem as a goat in the world.[1413.78] [1413.78][S01]And you can drive cars, you can jump on objects.[1416.373] [1416.373][S02] As a goat.[1417.29] [1417.29][S01] As a goat.[1417.89] [1417.89][S01]Yeah, you're a goat.[1418.97] [1418.97][S01]You can have wings and you can fly as well.[1421.57] [1421.57][S01]But it's a perfect sandbox game because you can[1424.03] [1424.03][S01]do so many different things.[1426.52] [1426.52][S01]And so in "Goat Simulator," there is a key which causes[1430.06] [1430.06][S01]the goat to become all floppy and ragdoll on the floor.[1433.45] [1433.45][S01]I'm not too sure what's the--[1435.202] [1435.202][S01]when you would actually as a player[1436.66] [1436.66][S01]use this, maybe to pretend.[1439.403] [1439.403][S02] Play dead.[1440.32] [1440.32][S01] Exactly.[1441.32] [1441.32][S01]But sometimes when we play test the agents,[1443.98] [1443.98][S01]it ragdolls on the floor, which brings[1448.0] [1448.0][S01]some funny moments in the team when we do the playtest.[1450.408] [1450.408][S02] Just expecting it to barge into a crowd of people[1452.95] [1452.95][S02]and it just ragdolls.[1453.83] [1453.83][S01] Yes.[1454.12] [1454.12][S01]Yes.[1454.62] [1454.62][S01]That can happen.[1455.287] [1455.287][S02] Can I see what it looks like?[1456.995] [1456.995][S01] Yeah, I'll show you.[1458.72] [1458.72][S01]So here, let's start by "Goat Simulator 3."[1464.217] [1464.217][S02] My fave.[1465.05] [1465.05][S01] So we have a video of the agent[1468.1] [1468.1][S01]here, which the instruction is to go to a green object.[1472.123] [1472.123][LAUGHTER][1473.62] [1474.97][S02] We've got a goat.[1476.44] [1476.44][S02]And we've got quite realistic environment.[1479.69] [1479.69][S02]And we've got the goat trotting forwards.[1481.977] [1481.977][S02]There's plenty of green, which the goat is walking past,[1484.31] [1484.31][S02]but it's doing a big jump into a pond.[1486.11] [1486.11][S02]And in the pond, there is an inflatable[1488.53] [1488.53][S02]in the shape of a snake, which the goat has successfully[1491.49] [1491.49][S02]reached.[1491.99] [1491.99][S01] It's green.[1493.115] [1493.115][S02] It's definitely green.[1494.66] [1494.66][S02]And it is an object.[1495.95] [1495.95][S01] It's an object, yes.[1496.55] [1496.55][S02] And it did a jump there to get there.[1498.65] [1498.65][S01] Yeah.[1499.525] [1499.525][S01]So sometimes the agent picks up on some human peculiarities.[1503.43] [1503.43][S01]For example, humans, when they play games,[1505.51] [1505.51][S01]are often bored to take the longest path.[1509.3] [1509.3][S01]For example, climbing up the stairs.[1510.8] [1510.8][S01]If you have a jetpack, you're just[1512.05] [1512.05][S01]going to jetpack over the stairs.[1513.5] [1513.5][S01]So one of the first agents where we asked,[1516.04] [1516.04][S01]get up to the platform, instead of taking[1518.26] [1518.26][S01]the stairs the long way, the agent[1519.97] [1519.97][S01]just jetpacked over the stairs.[1522.37] [1522.37][S01]So it does learn about human behavior.[1525.973] [1525.973][S02] The strange shortcuts.[1527.39] [1527.39][S02]I also noticed that rather than just walking into the pond,[1529.89] [1529.89][S02]it jumped, which is exactly what I would do as a human.[1532.68] [1532.68][S01] Yeah.[1533.93] [1533.93][S02] You want to make a splash.[1535.02] [1535.02][S01] Yes.[1535.4] [1535.4][S02] OK.[1535.89] [1535.89][S02]All right.[1536.13] [1536.13][S02]Well, I mean, it succeeded.[1537.255] [1537.255][S02]It succeeded.[1538.08] [1538.08][S01] So this is the game "Valheim,"[1541.04] [1541.04][S01]which is a Viking survival sandbox game.[1544.74] [1544.74][S01]And the task here that we asked the agent to carry out[1547.46] [1547.46][S01]is to pick up mushrooms.[1549.5] [1549.5][S02] OK, so again, another very beautifully crafted[1553.91] [1553.91][S02]environment.[1554.82] [1554.82][S02]The camera is moving around, which[1556.963] [1556.963][S02]I guess is the agent doing that, moving the camera around.[1559.38] [1559.38][S02]It has found quite quickly some mushrooms, shot them out[1562.745] [1562.745][S02]of the ground--[1563.37] [1563.37][S02]I don't know whether the Vikings had that particular ability--[1567.95] [1567.95][S02]and just carrying on wandering through the forest.[1570.63] [1570.63][S01] Yeah.[1570.86] [1570.86][S02] OK.[1571.485] [1571.485][S02]So definitely like camera control and movement in that 3D[1574.01] [1574.01][S02]environment, it's mastered.[1575.517] [1575.517][S01] Yes.[1576.35] [1576.35][S01]So I think because all the games that we use are 3D games,[1581.4] [1581.4][S01]the camera control is very similar from all those games.[1584.82] [1584.82][S01]So I there is a lot of experience[1586.378] [1586.378][S01]to learn from in terms of controlling the camera.[1588.42] [1588.42][S01]You can see that the movement of the camera is human-like.[1592.14] [1592.14][S01]So it hesitates.[1593.33] [1593.33][S01]It's not very robotic, just aim on the mushroom and go forward.[1596.08] [1596.08][S01]And that's because it's trained from human data.[1598.08] [1598.08][S01]If you train an agent with reinforcement[1600.41] [1600.41][S01]learning to do the same task, you could detect--[1602.99] [1602.99][S02] As opposed to imitation learning?[1604.22] [1604.22][S01] As opposed to imitation learning,[1606.262] [1606.262][S01]the agent would have very linear and robotic camera motion.[1610.83] [1610.83][S01]It would just go to the mushroom.[1612.47] [1612.47][S01]It needs to optimize its reward.[1614.43] [1614.43][S01]So it will not spend any time hesitating.[1617.43] [1617.43][S02] When you're instructing the agent,[1619.71] [1619.71][S02]you're not explaining what a mushroom is[1621.59] [1621.59][S02]or what a mushroom might look like in the game?[1623.79] [1623.79][S01] Yeah, so it gets that from the training data.[1626.58] [1626.58][S01]So in this training data, there is a bunch[1630.35] [1630.35][S01]of sequences of humans interacting with mushrooms,[1633.24] [1633.24][S01]picking up mushrooms.[1634.76] [1634.76][S01]And so it just learns from the data.[1637.02] [1637.02][S01]So an interesting task we could ask the agent here[1641.42] [1641.42][S01]is, if you hold out, if the agent has never[1643.85] [1643.85][S01]seen mushrooms or maybe never seen red mushrooms,[1646.65] [1646.65][S01]but maybe it has seen blue mushrooms,[1648.365] [1648.365][S01]and you ask the agent to pick up the red mushrooms,[1650.49] [1650.49][S01]is the agent able to generalize color and object type?[1655.547] [1655.547][S01]So those type of questions are the questions[1657.38] [1657.38][S01]we are interested in.[1658.58] [1658.58][S02] Which is definitely demonstrating[1660.47] [1660.47][S02]a sort of conceptual understanding of the environment[1662.33] [1662.33][S02]that it's in.[1662.97] [1662.97][S01] Yeah, it would.[1664.41] [1664.41][S01]Yes.[1665.39] [1665.39][S01]We're looking for this generality and transfer[1667.55] [1667.55][S01]between concepts and trying to put things together,[1670.65] [1670.65][S01]new concepts together to carry out a task.[1673.98] [1673.98][S01]And yeah, this will show understanding[1676.73] [1676.73][S01]of what a mushroom is and what the color red is.[1679.048] [1679.048][S02] In a way, the way that you describe the imitation[1681.59] [1681.59][S02]part of it, the imitation learning,[1683.42] [1683.42][S02]it sounds a little bit like you're[1685.49] [1685.49][S02]doing a sort of computer game version of predictive text,[1688.26] [1688.26][S02]almost.[1688.76] [1688.76][S02]Like, what has been done in this environment before,[1691.17] [1691.17][S02]how do you repeat it?[1692.682] [1692.682][S01] It is exactly that.[1694.14] [1694.14][S02] Is it?[1694.94] [1694.94][S01] Yes, so when you train language models[1697.52] [1697.52][S01]on language, the objective in which you train,[1700.8] [1700.8][S01]at least the initial model, is to predict[1703.85] [1703.85][S01]what's the next token.[1707.54] [1707.54][S01]I mean, if it's assuming that it's all human-generated text,[1711.44] [1711.44][S01]what would the human say next?[1713.16] [1713.16][S01]What word would the human say next?[1714.69] [1714.69][S01]Which is very similar to this, which is,[1716.55] [1716.55][S01]what would the human do next in terms of the keyboard and mouse[1720.17] [1720.17][S01]action?[1720.84] [1720.84][S01]So you're right, it is pretty much the same technique.[1724.245] [1724.245][S02] The same ideas.[1725.37] [1725.37][S02]But the key difference here is that you're instructing it.[1727.83] [1727.83][S02]You're saying, this is the objective for you[1730.55] [1730.55][S02]and go and do that in a way that a human would do?[1733.017] [1733.017][S01] Yes.[1733.85] [1733.85][S01]So it's given this instruction, what would the human do next?[1736.59] [1736.59][S01]But the language models are also,[1738.96] [1738.96][S01]in a way, prompted with some prior text.[1740.8] [1740.8][S01]So they would say, given that this text is[1742.55] [1742.55][S01]in your prompt, what is the next most probable word that you[1747.68] [1747.68][S01]should output?[1748.56] [1748.56][S01]So it's not very different.[1751.5] [1751.5][S01]The maybe infrastructure and architecture of the agent[1755.18] [1755.18][S01]itself is a bit different, because we here[1757.52] [1757.52][S01]need to produce a very different modality, which is action[1762.98] [1762.98][S01]in keyboard and mouse space.[1764.34] [1764.34][S02] It sounds like it's kind[1764.99] [1764.99][S02]of exploring the network of actions[1766.58] [1766.58][S02]that people have taken in the past.[1768.65] [1768.65][S02]Does it do original things, then?[1771.18] [1771.18][S01] So it will not do anything[1773.51] [1773.51][S01]that humans haven't done, per se,[1776.76] [1776.76][S01]but the agent is able to generalize[1778.76] [1778.76][S01]to unseen environments, so in a way that's kind of novel.[1782.18] [1782.18][S01]So it's like, you're a tennis player.[1785.12] [1785.12][S01]And suddenly, you're asked to do a badminton match,[1788.1] [1788.1][S01]right, as a human.[1788.93] [1788.93][S01]Because you've learned tennis, you[1790.61] [1790.61][S01]have some reflexes and mechanics that you know.[1794.17] [1794.17][S01]You know how to hold the racket, how to strike a ball.[1799.05] [1799.05][S01]And so you're going to do fairly well at badminton.[1802.02] [1802.02][S01]Not the best, but you're not going to be beginner level.[1805.08] [1805.08][S01]And in our agent, we see the same.[1807.24] [1807.24][S01]So what we've done is that we trained the agent[1809.99] [1809.99][S01]on all environments except one, and then we[1812.185] [1812.185][S01]test the agent on that one environment.[1813.81] [1813.81][S01]And we can see that the agent is able to do[1817.71] [1817.71][S01]a few things at a reasonable level in that environment.[1820.66] [1820.66][S01]So in a way, I don't know if you can say those are novel,[1824.62] [1824.62][S01]but it definitely shows some promise with respect[1827.28] [1827.28][S01]to generalization and generalizing to unseen[1829.56] [1829.56][S01]environments.[1830.52] [1830.52][S02] So, OK, tell me how well does it perform overall,[1833.19] [1833.19][S02]then?[1833.38] [1833.38][S01] Maybe I can tell you[1833.91] [1833.91][S01]about two interesting results.[1835.0] [1835.0][S02] Yes, please.[1836.0] [1836.0][S01] So first of all, we[1837.87] [1837.87][S01]observed that when we train an agent on all[1840.39] [1840.39][S01]of the environments, it performs better[1842.85] [1842.85][S01]than an agent that is only trained[1844.47] [1844.47][S01]on each singular individual environment.[1846.803] [1846.803][S02] So you train an agent on "No Man's Sky,"[1848.97] [1848.97][S02]"Goat Simulator 3," "Valheim," and then it does better[1853.74] [1853.74][S02]in, for example, "Goat Simulator 3"[1857.4] [1857.4][S02]than an agent that is trained only in "Goat Simulator 3?"[1860.452] [1860.452][S01] Exactly, yes.[1861.66] [1861.66][S01]So that was a very good result for us[1866.64] [1866.64][S01]because it demonstrates that the experience that you[1869.85] [1869.85][S01]get from training on variety of game[1872.13] [1872.13][S01]make your skill in each individual game stronger.[1875.97] [1875.97][S01]So what I like to think of is, let's say as a human,[1879.34] [1879.34][S01]maybe to go back to the tennis example, maybe I train on tennis[1882.15] [1882.15][S01]and I train on badminton.[1883.35] [1883.35][S01]And maybe I'll be better at both of those compared[1887.82] [1887.82][S01]to if I had only trained on, say, tennis[1890.73] [1890.73][S01]because maybe I gained some extra skills and subtlety[1893.85] [1893.85][S01]in the type of actions I need to do from being exposed to a wider[1898.56] [1898.56][S01]range of skills and experience.[1900.433] [1900.433][S02] Is it also partly they're just exposed to more[1902.85] [1902.85][S02]data, so more of how humans manipulate 3D environments,[1906.4] [1906.4][S02]more of different kinds of things that humans might do?[1909.07] [1909.07][S01] Yeah.[1909.945] [1909.945][S01]So games, different games offer different situations.[1912.55] [1912.55][S01]So the agent can extract knowledge from those situations[1916.11] [1916.11][S01]and maybe relate them to other situations from other games.[1919.54] [1919.54][S01]And so it just accumulates and consolidates the behavior[1922.86] [1922.86][S01]into something that is better than not having seen[1927.64] [1927.64][S01]this other experience, which in a way, our hypothesis when we[1931.41] [1931.41][S01]started the project was the same hypothesis as the language[1934.78] [1934.78][S01]model.[1935.28] [1935.28][S01]Training a lot of data will make some interesting properties[1940.05] [1940.05][S01]emerge, such as being able to generalize and transfer.[1943.72] [1943.72][S01]And so, yeah, that's what we also observed here.[1948.413] [1948.413][S02] But then I think also,[1949.83] [1949.83][S02]that result isn't immediately obvious because it could well[1952.707] [1952.707][S02]have been the case that actually all the things that it learns[1955.29] [1955.29][S02]from "No Man's Sky," like, confuse it when it goes back[1958.29] [1958.29][S02]to "Goat Simulator 3."[1959.29] [1959.29][S01] Yes.[1960.123] [1960.123][S01]So we were worried about that because we thought maybe[1962.49] [1962.49][S01]there will be a destructive interference between games.[1965.98] [1965.98][S01]That's true.[1967.23] [1967.23][S01]We picked those games to all be 3D because we thought there was[1972.33] [1972.33][S01]a chance that some of the experience would transfer[1975.297] [1975.297][S01]between games.[1975.88] [1975.88][S01]If we had trained on "Tetris" and "No Man's Sky,"[1978.04] [1978.04][S01]we wouldn't expect any positive interference here.[1983.362] [1983.362][S02] Slightly different environment, isn't it?[1985.57] [1985.57][S01] Yes.[1986.403] [1986.403][S01]But because all of those games have some similar concepts,[1990.33] [1990.33][S01]same way of navigating around, moving the camera,[1993.53] [1993.53][S01]we thought maybe there would be some interesting and positive[1999.16] [1999.16][S01]outcome to training on multiple games.[2002.917] [2002.917][S02] You told me that there were two,[2004.75] [2004.75][S02]there were two big results.[2005.77] [2005.77][S01] Yes.[2006.45] [2006.45][S01]So the other game, the other result,[2008.02] [2008.02][S01]sorry, which I mentioned before, was[2010.68] [2010.68][S01]that when you train an agent on all environments but one,[2015.66] [2015.66][S01]it performs roughly as well on this held-out environment[2020.73] [2020.73][S01]as the agent that has only been trained on that environment.[2023.445] [2023.445][S02] OK, wait, wait, wait.[2024.82] [2024.82][S02]So, right, you've got eight games.[2027.6] [2027.6][S02]You train an agent on them and leave off "Goat Simulator 3."[2031.98] [2031.98][S02]It's never seen "Goat Simulator 3" before.[2034.86] [2034.86][S02]And then you compare it against an agent that's seen nothing[2039.39] [2039.39][S02]else but "Goat Simulator 3."[2040.557] [2040.557][S01] Exactly.[2041.557] [2041.557][S02] And then you test them in "Goat Simulator 3."[2044.17] [2044.17][S01] Yes, and the SIMA agent here[2046.89] [2046.89][S01]performs nearly as well as the environment.[2049.445] [2049.445][S01]We call it the environment expert[2050.82] [2050.82][S01]because it's an expert in goat expert.[2053.409] [2053.409][S02] Goal expert, sure.[2055.23] [2055.23][S01] So, yes, it doesn't[2056.94] [2056.94][S01]perform as well, on average, but it does perform reasonably well.[2062.83] [2062.83][S01]So it has learned to generalize to unseen.[2065.79] [2065.79][S01]All the other games have different avatars, for example.[2068.73] [2068.73][S01]The agent has not--[2069.69] [2069.69][S02] There's no goats anywhere else.[2071.482] [2071.482][S02]There's no goats in space.[2072.788] [2072.788][S01] But the agent is able to move the goat around[2075.33] [2075.33][S01]and perform a few tasks.[2076.989] [2076.989][S01]Of course, there are some actions in "Goat Simulator"[2080.489] [2080.489][S01]which are specific and unique to "Goat Simulator,"[2082.86] [2082.86][S01]and that we don't--[2083.739] [2083.739][S02] There's no ragdolling astronauts.[2085.0] [2085.0][S01] No, and we don't expect the agent[2087.042] [2087.042][S01]to be able to get that because it has never seen it,[2090.81] [2090.81][S01]or it has never seen experience from it.[2093.659] [2093.659][S01]But other things like navigation, maybe concepts[2097.44] [2097.44][S01]about objects and colors, that is something[2101.49] [2101.49][S01]that it can achieve.[2102.348] [2102.348][S02] So how do you expand that[2103.89] [2103.89][S02]beyond these sort of tasks that last[2105.96] [2105.96][S02]for 10 seconds, 15 seconds of jumping and picking up[2109.14] [2109.14][S02]a mushroom?[2109.74] [2109.74][S01] We are working currently[2111.407] [2111.407][S01]on trying to find ways of doing that.[2113.96] [2113.96][S01]First of all, there is a lot to improve because our agent is not[2118.87] [2118.87][S01]at human level, even for those short horizon tasks.[2122.53] [2122.53][S01]For longer horizon tasks, we are looking into methods which[2129.76] [2129.76][S01]would allow us to do that.[2131.59] [2131.59][S01]You mentioned large language models before,[2134.15] [2134.15][S01]which are able to maybe reason on a higher time horizon, longer[2138.94] [2138.94][S01]time horizon.[2139.88] [2139.88][S01]So maybe combining those models with our agent could be a way,[2143.44] [2143.44][S01]but that's future work.[2144.562] [2144.562][S02] Can I play alongside one of these things,[2146.77] [2146.77][S02]just to make my experience of the game better?[2148.91] [2148.91][S02]Can I cheat, basically?[2150.315] [2150.315][S01] So some agents do[2151.69] [2151.69][S01]find ways of exploiting the games to do things that were not[2158.02] [2158.02][S01]originally intended.[2159.17] [2159.17][S01]So in a way, you could call that cheating, I guess.[2162.82] [2162.82][S01]But playing, like, once we have very competent AI systems, which[2168.873] [2168.873][S01]could do all sorts of things, playing games with you[2171.04] [2171.04][S01]could also be one of the things you[2172.54] [2172.54][S01]want to do with those systems.[2175.12] [2175.12][S01]That would be quite fun, I think, especially[2177.45] [2177.45][S01]in these sandbox games.[2178.57] [2178.57][S01]If you had an agent, which you could say, OK,[2180.445] [2180.445][S01]now I want to play hide-and-seek in "Goat Simulator,"[2183.33] [2183.33][S01]and let's say it's multiplayer, you have two goats,[2185.91] [2185.91][S01]and you can play whatever you want.[2188.012] [2188.012][S01]I think it would be quite fun to interact[2189.72] [2189.72][S01]with an agent in exactly the way you want.[2192.55] [2192.55][S01]So the way we currently sometimes use large language[2196.11] [2196.11][S01]models is we prompt them to do all sorts of things,[2198.58] [2198.58][S01]write a poem, do some role play, write code.[2202.23] [2202.23][S01]If we had the same ability, but also[2203.88] [2203.88][S01]with an agent that has an avatar in game, which you could prompt,[2208.75] [2208.75][S01]I think that could be lots of fun for gamers.[2211.36] [2211.36][S02] Yeah, it's like, level, level,[2213.36] [2213.36][S02]level up on an NPC, isn't it?[2215.53] [2215.53][S01] Yes.[2216.43] [2216.43][S02] OK, so what is the big objective here, then?[2218.77] [2218.77][S01] So I think what we[2221.01] [2221.01][S01]are interested in in the SIMA team is[2225.03] [2225.03][S01]to build truly general agents.[2226.87] [2226.87][S01]So we use games as a platform to drive innovation[2231.93] [2231.93][S01]and breakthrough into this particular space[2234.49] [2234.49][S01]so that we can get a step closer to AGI[2236.96] [2236.96][S01]and understand how to build those truly general agents.[2239.99] [2239.99][S01]So we are really in for the research towards AGI.[2242.553] [2242.553][S02] What's the definition of AGI[2244.22] [2244.22][S02]that DeepMind work to?[2246.295] [2246.295][S01] I think the definition of AGI[2248.17] [2248.17][S01]is an agent that is as general and capable as a human.[2254.56] [2254.56][S01]And I think the general is key here.[2257.23] [2257.23][S01]It could be better than a human, but at least at human level.[2260.39] [2260.39][S02] So you can pick it up and drop it[2261.28] [2261.28][S02]into different environments and it[2262.697] [2262.697][S02]can do as well as a human can?[2264.38] [2264.38][S01] Yes.[2265.213] [2265.213][S01]So, I think if you think about gaming, as a human,[2268.868] [2268.868][S01]maybe there is a new game coming out.[2270.41] [2270.41][S01]I install it and very quickly I will be able to play this game.[2274.48] [2274.48][S01]Maybe I need to go through the tutorial[2276.13] [2276.13][S01]because there are some specificity to the game that I[2278.38] [2278.38][S01]need to learn, but in general, I think[2280.54] [2280.54][S01]we are very good at adapting to those new situations,[2283.13] [2283.13][S01]to those new environments, and be competent at it.[2285.77] [2285.77][S01]And we really want the same for our agents.[2287.953] [2287.953][S01]We want to drop the agent in a new environment[2289.87] [2289.87][S01]or show it a new game and observe that they can, indeed,[2294.31] [2294.31][S01]also adapt and become competent at those new situations.[2297.753] [2297.753][S02] Well, you've sort of done that already, haven't you?[2300.42] [2300.42][S01] There is still a long way to go.[2302.42] [2302.42][S01]So, our agents have been trained to only perform simple actions,[2307.89] [2307.89][S01]but really, there is so much more skills and scenarios[2315.5] [2315.5][S01]which we could use to train our agents.[2317.49] [2317.49][S01]How do we make an agent that could handle 1,000 different[2322.25] [2322.25][S01]games?[2323.15] [2323.15][S01]What does it take?[2324.8] [2324.8][S01]There are some breakthroughs that needs to happen.[2327.035] [2327.035][S02] So then let's say that you manage[2328.91] [2328.91][S02]that, which I'm sure you will.[2331.71] [2331.71][S02]What happens then?[2334.692] [2334.692][S02]Is the idea that you can take that learning and then apply[2338.48] [2338.48][S02]it to really generalist agents that exist in the environment[2342.44] [2342.44][S02]that we all live in?[2343.7] [2343.7][S01] Yes, there are methods.[2345.41] [2345.41][S01]How did we build the agent?[2346.83] [2346.83][S01]I think the understanding what method[2348.44] [2348.44][S01]we use to achieve this result is very important, that we can then[2351.32] [2351.32][S01]apply the same method to other domains, which[2354.37] [2354.37][S01]we might be interested in.[2355.61] [2355.61][S01]Understanding technical details of how to build those agents,[2361.05] [2361.05][S01]understanding how to collect the data, which data is useful,[2365.72] [2365.72][S01]algorithms, how do we train the agent,[2368.27] [2368.27][S01]I think that's very important as well.[2371.29] [2371.29][S01]So, yeah, any kind of innovation and progress[2373.87] [2373.87][S01]we can make towards understanding the[2375.73] [2375.73][S01]how to make progress in AGI is what we are after.[2380.505] [2380.505][S02] But these are lessons[2381.88] [2381.88][S02]that can apply elsewhere?[2383.467] [2383.467][S01] They could apply to other domains.[2385.55] [2385.55][S01]Yes.[2385.6] [2385.6][S02] OK.[2386.225] [2386.225][S02]So, give me an example, then.[2388.87] [2388.87][S02]What kind of agents should I be hoping for?[2392.8] [2392.8][S01] I mean, so are we talking about the far future?[2395.51] [2395.51][S02] Yeah, go on.[2396.51] [2396.51][S01] OK, so I think I could say an example of agents[2401.98] [2401.98][S01]would be self-driving cars that are safe and reliable.[2405.88] [2405.88][S01]I think those are agents which take[2408.64] [2408.64][S01]actions that have consequences.[2411.08] [2411.08][S01]And so having very robust and reliable self-driving car, which[2415.99] [2415.99][S01]can generalize to unseen circumstances,[2418.9] [2418.9][S01]would be very important.[2419.93] [2419.93][S01]And I think the generality here is very important.[2422.055] [2422.055][S01]Because when you drive, you never[2423.43] [2423.43][S01]know what's going to happen.[2424.597] [2424.597][S01]I mean, if we had agents in the real world,[2427.15] [2427.15][S01]one of the things we would like would be for agents to carry out[2431.47] [2431.47][S01]instructions for us.[2432.38] [2432.38][S01]For example, would be a robot in your house[2435.64] [2435.64][S01]where you say, maybe carry these heavy objects to the table.[2440.54] [2440.54][S01]That's a task which is defined in language,[2443.64] [2443.64][S01]which would be very useful.[2444.8] [2444.8][S01]Maybe virtual agents, which can code and write software[2450.32] [2450.32][S01]would be very useful for us.[2453.38] [2453.38][S01]Maybe agents that can help you in everyday tasks which you do[2458.66] [2458.66][S01]online, for example, shopping.[2461.27] [2461.27][S01]I mean, we spend a lot of time researching what to buy.[2465.3] [2465.3][S01]If I want a new pair of headphones,[2467.61] [2467.61][S01]I might spend hours to understand[2469.58] [2469.58][S01]which one is good for me.[2471.15] [2471.15][S01]But if I could have an agent that[2472.76] [2472.76][S01]can do this research for me, that would be very useful,[2476.76] [2476.76][S01]I think.[2477.26] [2477.26][S02] I'm just thinking about,[2478.19] [2478.19][S02]I'm just going back to your AGI stuff, actually, if I may,[2480.607] [2480.607][S02]just to finish on.[2481.92] [2481.92][S02]Do you think that the sparks of AGI will happen in your lab?[2486.76] [2486.76][S01] Well, it's very difficult to say,[2489.02] [2489.02][S01]but we do hope, yes, that the methods we develop[2493.45] [2493.45][S01]will help to make progress towards AGI.[2497.32] [2497.32][S01]I think if we had an agent that could play any games[2502.84] [2502.84][S01]and carry out any instruction in any game--[2505.66] [2505.66][S02] And perform a superhuman ability?[2507.64] [2507.64][S01] Yes, I think that would be a good milestone[2513.38] [2513.38][S01]to achieve because that would show human-level intelligence,[2516.99] [2516.99][S01]at least in games.[2518.51] [2518.51][S01]But games are very rich and sometimes are[2522.057] [2522.057][S01]a bit like the real world.[2523.14] [2523.14][S01]You need to understand your surroundings.[2525.058] [2525.058][S01]You need to understand objects.[2526.35] [2526.35][S01]You need to understand the consequence of your actions.[2529.28] [2529.28][S01]Especially if you add language into the mix[2531.24] [2531.24][S01]because if you add language into the mix for tasks, then really[2535.01] [2535.01][S01]you could ask the agent to do anything.[2536.67] [2536.67][S01]So if such agent is able to carry out[2538.28] [2538.28][S01]any instruction in any game, I think[2540.29] [2540.29][S01]it would be one step closer to AGI.[2543.262] [2543.262][S02] Fred, thank you very much for joining me.[2545.47] [2545.47][S01] Thank you so much for having me.[2547.61] [2547.61][S02] There have been a few moments now[2549.485] [2549.485][S02]while making this podcast where we have been here[2551.66] [2551.66][S02]at the early stages of a project, before the results were[2555.41] [2555.41][S02]grand or headline grabbing, when there was only the tiniest[2558.38] [2558.38][S02]little spark of something to see.[2560.36] [2560.36][S02]Like, remember when we first visited the robot lab[2563.09] [2563.09][S02]to watch them flailing around in circles, or the first time[2566.78] [2566.78][S02]that we saw AlphaFold when it was only just starting[2570.59] [2570.59][S02]to unravel the mysteries of proteins?[2573.4] [2573.4][S02]Well, I think it feels a lot like this is[2575.61] [2575.61][S02]the stage we're at with SIMA.[2577.78] [2577.78][S02]Because sure, picking up a mushroom[2579.9] [2579.9][S02]isn't quite the vision of the AI future[2582.09] [2582.09][S02]that we've been waiting for, but it's the direction[2585.15] [2585.15][S02]that those agents are moving in that[2588.12] [2588.12][S02]really holds the secret promise to what might be possible.[2591.1] [2591.1][S02]Because if we want AI that can make its own decisions,[2594.79] [2594.79][S02]achieve its own objectives, operate independently[2598.65] [2598.65][S02]of our instructions, then this is a vital step along the way.[2603.76] [2603.76][S02]So today, it might be a mushroom.[2606.24] [2606.24][S02]The question is, what will it be next time we come back?[2610.18] [2610.18][S02]You've been listening to "Google DeepMind--[2612.07] [2612.07][S02]The Podcast," with me, Professor Hannah Fry.[2614.73] [2614.73][S02]If you like what you just heard, hit Subscribe on YouTube[2618.09] [2618.09][S02]or follow us on your favorite podcast platform.[2621.29] [2621.29][S02]We've got an exciting line up of conversations to come,[2623.65] [2623.65][S02]including an interview with Pushmeet Kohli, who[2627.0] [2627.0][S02]oversees Science Research at Google DeepMind.[2629.95] [2629.95][S02]If you have someone in mind that you'd[2631.89] [2631.89][S02]love to hear from on this podcast, do leave us a comment.[2635.37] [2635.37][S02]See you next time.[2636.5] [2636.5][MUSIC PLAYING][2640.75]