[11.512][S01] The idea of creating artificial creatures[14.081] [14.081][S01]has obsessed us for millennia,[16.683] [16.683][S01]but for the purposes of this exercise before we go any further,[19.953] [19.953][S01]I’m going to ask you to purge any of the following thoughts from your head.[24.458] [24.458][S01]Checklist: Hephaestus expelled from Olympus[27.661] [27.661][S01]and then built two servant robots,[29.63] [29.63][S01]the Bhuta Vahana Yanta or spirit movement machines[32.966] [32.966][S01]of 12th Century India made to protect the relics of Buddha,[36.403] [36.403][S01]the chess playing mechanical Turk, just a bloke in a box pulling levers.[40.874] [40.874][S01]Mary Shelley’s Frankenstein - he walks, he talks, not a lot else.[45.212] [45.212][S01]Kubrick’s HAL “Don’t Call Me Dave,” C3P0, R2D2, canine, NS2,[50.684] [50.684][S01]I mean seriously these are just numbers and letters.[53.32] [53.32][S01]Robbie the Robot, robot lovers, RoboCop.[56.924] [62.429][S01]Feeling better? Okay. Let’s get going.[68.669] [71.572][S01]I’m Hannah Fry, I’m an Associate Professor in Mathematics,[74.274] [74.274][S01]and I am AI curious, and this is DeepMind,[78.846] [78.846][S01]the podcast series where we look[80.347] [80.347][S01]at the fast-moving story of artificial intelligence.[84.051] [84.051][S01]We’ve been talking to the scientists,[85.552] [85.552][S01]researchers and engineers based at DeepMind in London.[89.489] [89.489][S01]We’re looking at how they’re approaching the science of AI.[92.96] [92.96][S01]And some of the tricky decisions[94.595] [94.595][S01]the whole field is wrestling with at the moment.[98.198] [98.198][S01]So whether you just want to know more[100.2] [100.2][S01]or want to be inspired on your own AI journey,[103.303] [103.303][S01]then this is the place to be.[105.739] [108.675][S01]You see the thing is - robots sell. We’ve long lusted after the idea[113.714] [113.714][S01]of upending the natural order with human ingenuity.[117.117] [117.117][S01]We just can’t seem to leave it alone,[120.053] [120.053][S01]and in this episode we are looking at AI and robotics.[124.258] [124.258][S01]Murray Shanahan is the Senior Scientist at DeepMind -[126.527] [126.527][S01]he’s also a professor of cognitive robotics[129.696] [129.696][S01]at Imperial College London, and growing up,[132.466] [132.466][S01]Murray was utterly mesmerized by science fiction,[136.47] [136.47][S01]so you can picture his face when the Hollywood film director Alex Garland[140.674] [140.674][S01]approached him following the publication of his book[143.51] [143.51][S01]Embodiment and the Inner Life.[145.979] [145.979][S06] Alex contacted me and said: “Oh I’m writing a script for ah film[150.05] [150.05][S06]about AI and consciousness and I read your book[152.519] [152.519][S06]and it you know helped to crystallize some ideas[154.655] [154.655][S06]and would you like to chat about ah chat about it?[157.391] [157.391][S06]And ah and so of course it was you know it was a great opportunity[161.695] [161.695][S06]ah to get involved in the science fiction film and then[164.364] [164.364][S06]and then to my great good fortune it turns out to be an absolute cracker.[168.235] [169.203][S01] And that is how Murray became scientific advisor[173.04] [173.04][S01]on the Oscar-winning film Ex Machina.[176.176] [176.844][S01]I met with Murray to get a potted history of AI.[180.08] [180.08][S01]Murray, people tend to think of AI as this this this very new thing,[183.884] [183.884][S01]a very a very modern invention,[185.886] [185.886][S01]but it’s actually been around for quite a long time.[188.121] [188.121][S06] It has. The idea of artificial intelligence,[190.557] [190.557][S06]the idea of making artificial creatures dates back to Greek mythology[194.561] [194.561][S06]but the sort of modern conception of AI perhaps really dates back to Alan[199.766] [199.766][S06]Turing’s paper published in the 1950s[202.936] [202.936][S06]where he first kind of asked the question -[205.639] [205.639][S06]could a machine think?[207.14] [207.14][S06]And gave a number of kind of refutations for counter arguments[210.477] [210.477][S06]to the idea that a machine could think.[212.379] [212.379][S01] This is where the Turing test comes from.[214.615] [214.615][S06] And this was this famous paper inaugurated the so-called[217.751] [217.751][S06]Turing test because Turing didn’t call it[219.319] [219.319][S06]the Turing test [Hannah laughs][220.554] [220.554][S06]which is the idea that we should subject a machine[223.257] [223.257][S06]to a test to see whether it’s basically -[225.125] [225.125][S06]whether it’s indistinguishable from a human in dialogue.[228.228] [228.228][S06]The term “artificial intelligence” was actually coined by John McCarthy,[232.299] [232.299][S06]a Stanford professor - he was at MIT at the time,[234.801] [234.801][S06]and John McCarthy organized a conference in 1956[239.106] [239.106][S06]bringing together a lot of leading thinkers in maths[241.542] [241.542][S06]to try and scope out the idea of building a thinking machine[244.945] [244.945][S06]and he coined the term artificial intelligence.[247.247] [247.247][S01] What did they describe it as at that time -[250.217] [250.217][S01]how did they see artificial intelligence?[252.686] [252.686][S06] John McCarthy in particular - his idea of artificial intelligence[256.79] [256.79][S06]that he had in mind was a kind of system that would answer questions really[260.894] [260.894][S06]and be able to engage in dialogue with humans, so…[264.431] [264.431][S01] Something that you are actually just talking to -[265.799] [265.799][S06] So something that you’re that you’re talking to[267.501] [267.501][S06]although of course in those days it wouldn’t have been through speech,[270.304] [270.304][S06]it would have been by typing in at the keyboard,[272.439] [272.439][S06]and it was very much a disembodied notion of artificial intelligence,[275.909] [275.909][S06]so this system didn’t have a body and interact with the physical world[280.414] [280.414][S06]in the way that we do or animals do or indeed that robots do[283.217] [283.217][S06]so they weren’t really thinking about robotics at that point.[285.953] [285.953][S01] I’m sort of imagining something like um HAL in “2001 Space Odyssey,”[289.756] [289.756][S01]except typing in rather than speaking to -[291.658] [291.658][S06] Yeah, and a kind of nice version, you know.[294.027] [294.027][S06]That their approach to artificial intelligence[296.163] [296.163][S06]was to build systems that reasoned in logic[299.366] [299.366][S06]and we now think of that whole sort of approach[301.869] [301.869][S06]to artificial intelligence of using logic and reasoning[304.938] [304.938][S06]as so-called good old fashioned artificial intelligence,[307.641] [307.641][S06]or Gofi, or classical AI.[310.944] [311.512][S01] Now I know, I know I did a bad thing there.[313.347] [313.347][S01]I mentioned HAL but nobody ever said this was going to be easy.[317.05] [317.05][S01]But to recap - in classical AI,[319.453] [319.453][S01]you have to write down a complete list of rules[321.755] [321.755][S01]for how you want your agent to think.[324.558] [324.558][S01]If this happens, then do this. If that happens, then do that.[328.829] [328.829][S01]It’s a nice idea in theory, but if your agent is going to know[331.665] [331.665][S01]how to handle every possible scenario you could throw at it,[335.836] [335.836][S01]it’s going to need to be a long, long list.[339.973] [339.973][S06] There was a project called Psych[342.342] [342.342][S06]which attempted to write out all the rules of common sense[346.68] [346.68][S06]to build an enormous encyclopedic database of common sense.[350.017] [350.017][S01] Can you remember any of them?[351.018] [351.018][S06] Well I mean there would be things like -[352.419] [352.419][S06]if you’ve got a container, and you put something in that container[357.191] [357.191][S06]and then move that container somewhere else,[359.359] [359.359][S06]then the thing that was in it gets moved as well.[361.862] [361.862]You know - [S01] Common sense![362.729] [362.729][S06] Yeah, stuff like that you know,[364.364] [364.364][S06]and if you buy something and pay for it,[367.134] [367.134][S06]you’ll have less money than you had in the first place.[369.937] [369.937][S06]So do I think it’s impossible to do that?[372.239] [372.239][S06]I think it’s impossible in practice.[374.174] [374.174][S06]Because it turns out that the sheer number of rules[377.244] [377.244][S06]that you would have to write is absolutely enormous.[379.813] [379.813][S01] We might have moved away from this long list of rules by now[382.749] [382.749][S01]as a way to teach our artificial intelligence in favor of agents[386.52] [386.52][S01]that can learn the rules for themselves. But the skills that we want our AI[391.258] [391.258][S01]to have like good old fashioned common sense[395.295] [395.295][S01]are just as important now as they ever were.[398.665] [402.402][S01]Imagine one day long into the future a wealthy computer scientist[406.74] [406.74][S01]builds an AI to manage his stamp collection.[410.377] [410.377][S01]He plugs it into the internet, gives it access to his bank account[413.847] [413.847][S01]and sets up the challenge to buy as many stamps as possible.[418.051] [418.051][S01]At first the agent acts as its creator intended,[421.321] [421.321][S01]signing up to eBay, bidding on stamps. But after a while he gets another idea.[427.461] [430.597][S01]More money equals more stamps,[433.433] [433.433][S01]so why not start trading on the stock market to make more money?[437.504] [437.504][S01]And it soon realizes it can get the stamps cheaper[440.607] [440.607][S01]if it can get them at source.[442.709] [442.709][S01]So the agent buys up a factory, converts its manufacturing process[446.58] [446.58][S01]to stamp-making, and goes on with achieving its goal.[450.684] [454.121][S01]But of course the limiting factor here is paper - more paper, more stamps.[459.293] [459.293][S01]So it starts commanding forests to be felled, the wood to be processed,[463.997] [463.997][S01]all to feed its single minded ambition more stamps.[469.036] [470.404][S01]Now there’s no denying that the AI is doing what it was told,[473.841] [473.841][S01]but it’s doing so at any cost,[476.743] [476.743][S01]and any agent without some kind of commons[479.413] [479.413][S01]sense will be at risk of taking our instructions a bit too literally.[485.319] [486.086][S01]This might be a bit of an extreme example, but Victoria Krakovna,[490.057] [490.057][S01]a research scientist a DeepMind working on AI safety[494.161] [494.161][S01]is already seeing agents[495.762] [495.762][S01]that aren’t exactly behaving in the way their designers intended.[500.234] [500.234][S04] A reinforcement agent that was playing a boat racing game[503.904] [503.904][S04]and the intended behavior there was to go around the race track[507.14] [507.14][S04]and finish the race as soon as possible, and the agent was encouraged to do this[511.345] [511.345][S04]by having these little green squares along the track[514.348] [514.348][S04]that would give it rewards,[515.616] [515.616][S04]and then what the agent figured out is that[517.184] [517.184][S04]instead of actually playing the game it could be going around in circles[520.153] [520.153][S04]and heading the same green squares over and over again to rack up more points,[523.624] [523.624][S04]and then you have this this whole situation with the boat going in circles[527.761] [527.761][S04]and crashing into everything and catching on fire[529.897] [529.897][S04]and still getting more points than to otherwise[531.832] [531.832][S04]get this this kind of situations are quite common.[534.368] [534.368][S01] But because you haven’t ever stopped the AI from doing that,[537.104] [537.104][S01]or told the AI that you don’t want it to do that,[539.573] [539.573][S01]that’s a perfectly sensible solution for it to come across.[543.143] [543.143][S04] Yeah, from the perspective of the AI[544.878] [544.878][S04]it can’t really tell that the solution is a cheat.[548.115] [548.115][S04]It’s just something that gives it a lot of rewards,[550.751] [550.751][S04]so it can’t necessarily distinguish between the general solution[554.555] [554.555][S04]and an overly creative solution that humans haven’t foreseen.[559.059] [560.694][S01] There are plenty of examples like this.[563.43] [563.43][S01]One team of researchers created an agent[565.832] [565.832][S01]inside a very simple two-dimension computer game[569.703] [569.703][S01]and tasked it with building itself a body[571.972] [571.972][S01]to get itself from the start line to the finish line.[575.108] [575.108][S01]Quite quickly it worked out that it could just build itself taller[579.012] [579.012][S01]and taller and taller[580.881] [580.881][S01]until it was as high as the track was long,[583.817] [583.817][S01]and then just flopped forwards to cross the line.[587.12] [587.12][S01]And there was the agent playing the game of Tetris[589.489] [589.489][S01]which realized it could just pause the game forever and never lose.[594.261] [594.261][S01]But there is a balancing act here. We don’t want our AI misbehaving,[598.365] [598.365][S01]but we also don’t want to restrict our agents too much.[601.602] [601.602][S04] And this is part of what’s tricky about achieving safe behaviors[605.639] [605.639][S04]that we don’t want to hamper the systems’ ability[608.675] [608.675][S04]to come up with really interesting and innovative solutions[611.578] [611.578][S04]that we have not seen - so we don’t just want human imitation.[615.749] [615.749][S04]We want super human capabilities but without unsafe behavior.[619.553] [622.189][S01] There is a very fine line between a naughty algorithm[625.959] [625.959][S01]and one that’s finding innovative solutions to problems[628.629] [628.629][S01]that humans haven’t been able to solve.[631.331] [631.331][S01]The AI doesn’t know the difference between the two -[634.701] [634.701][S01]it doesn’t understand what’s really important to us,[637.771] [637.771][S01]it doesn’t have any common sense -[640.407] [640.407][S01]and that means you have to be very careful[642.876] [642.876][S01]when you’re setting up incentives and rewards for your agents.[647.314] [647.314][S04] Part of the reason that this is so difficult is this general affect[651.285] [651.285][S04]that’s called Goodhart’s law in economics -[653.453] [653.453][S04]what Goodhart’s law comes when a metric becomes a target,[656.657] [656.657][S04]it ceases to be a good metric.[658.358] [660.427][S01] A classic example of Goodhart’s Law comes from British India,[663.897] [663.897][S01]when authorities offered cash rewards for dead cobras[667.935] [667.935][S01]as a way to decrease the population of snakes.[672.105] [672.105][S01]Unbeknownst to the British, the locals started to breed cobras[676.076] [676.076][S01]in order to take advantage of the reward.[679.179] [679.179][S01]As soon as the authorities found this out,[682.015] [682.015][S01]they scrapped the scheme altogether and revoked the rewards.[686.386] [686.386][S01]But now there were these snake farms everywhere filled with worthless cobras,[691.925] [691.925][S01]so what did the locals do? Release the cobras into the country,[695.796] [695.796][S01]resulting in an increase in the cobra population.[700.467] [700.467][S01]This is what the scientists call a specification problem -[704.938] [704.938][S01]when your specified objective fails to bring about the intended behavior.[710.444] [710.444][S01]This is generally quite likely to happen because human preferences[714.114] [714.114][S01]tend to be quite complex and whenever we try to distill them[717.117] [717.117][S01]into some kind of specification or something that we say we want[719.853] [719.853][S01]it’s going to be a lot simpler than our real preferences are[724.625] [724.625][S01]and it wouldn’t necessarily capture everything that’s important to us.[728.762] [731.131][S01]Let us imagine we’re living in the future[733.333] [733.333][S01]where robot butlers are commonplace.[736.904] [736.904][S01]You clearly specify the objective for your robot -[740.04] [740.04][S01]it should serve you at all times.[743.443] [743.443][S01]Now, how is that agent going to feel about its own off switch.[749.116] [749.116][S05] Your robot always has an incentive to preserve its own function.[753.854] [753.854][S05]If it gets turned off, it can’t vacuum the floors anymore.[757.224] [757.224][S05]It can’t bring you coffee. It want to disable its off switch for example.[761.762] [761.762][S01] Jan Leike is a Senior Research Scientist at DeepMind[764.665] [764.665][S01]also working on AI safety.[766.8] [766.8][S05] If you turn it off, then it can’t vacuum the floors.[769.536] [769.536][S05]So if it understands how the off switch the off switch mechanism works,[774.474] [774.474][S05]it would want to disable it. What we want is you want our systems[778.145] [778.145][S05]to actually do something that is good for us -[782.015] [782.015][S05]ready to do something that we actually wanted, not just what we said we wanted.[787.421] [787.421][S01] But how do you get around these kinds of problems?[790.357] [790.357][S01]Well we already know that writing a long list of dos and dont’s won’t work.[795.729] [795.729][S01]However long your list gets you’re always going to forget something.[800.334] [800.334][S01]The new breed of learning agents is going to need a different approach.[804.838] [804.838][S05] So one direction that we are pursuing is learning what functions[808.509] [808.509][S05]for reinforcement learning agents. And you can kind of think of this[811.712] [811.712][S05]as learning what your systems should be doing from human feedback.[817.284] [817.284][S05]So for example in in one work that we did together with Open AI[822.389] [822.389][S05]is we trained a simulated robot to do a backflip,[826.727] [826.727][S05]and um the way this works is like the robot does some movement[831.632] [831.632][S05]and then you watch a video of that movement and you or you say two videos[836.703] [836.703][S05]and then you can compare which of those looks more like a backflip.[840.374] [840.374][S01] Jan’s experiment has a human sitting and watching a screen,[844.244] [844.244][S01]looking at an AI attempting to do a backflip.[848.182] [848.182][S01]Each time the human will feedback on whether the attempt was good enough,[852.386] [852.386][S01]slowly nudging the agent in the right direction.[856.223] [856.223][S01]Here is the key idea - with constant human feedback,[860.827] [860.827][S01]the human can communicate their preferences[864.064] [864.064][S01]without having to actually specify them,[867.701] [867.701][S01]and risk oversimplifying things in the process.[871.104] [871.104][S05] And after like a few hundred rounds of feedback,[874.141] [874.141][S05]the robot can actually perform a backflip.[876.243] [876.243][S05]It’s kind of learn what the objective should be -[878.779] [878.779][S05]that the objective should be a backflip, and what a backflip is.[882.082] [882.082][S05]This is difficult to specify precisely what a backflip would be[887.487] [887.487][S05]in so in my case like I can’t do a backflip, right?[891.792] [891.792][S05]And but I can see if the systems do a backflip,[895.629] [895.629][S05]and in some ways like this allows us to get super human capability.[901.101] [901.101][S01] The AI then is essentially being rewarded[904.738] [904.738][S01]by pleasing the human - in a way?[907.407] [907.407][S05] Exactly. And so we can we can do a little experiment.[911.111] [911.144][S01] Okay.[912.079] [912.079][S05] I’ll try to teach you to make a sound.[914.047] [914.047][S01] Okay. [S05] By giving feedback.[916.483] [916.483]So you’ll make two sounds. [S01] Mm-hmm.[918.385] [918.385][S05] And I’ll tell you which of the two sounds is closer[921.054] [921.054][S05]to what I have in mind.[922.389] [922.389][S01] Okay. [S05] Yeah, does that sound good?[924.558] [924.558][S01] Let’s do it, let’s do it! So I’m being the AI here?[926.593] [926.593][S05] Yeah you’re being the AI and I’m being the human teacher.[929.329] [929.329][S01] And you’re training me essentially with reinforcement learning,[932.533] [932.533][S01]and my reward function is getting you to be happy.[935.469] [936.069][S05] You know that’s like - so the reward function here[938.172] [938.172][S05]is like something is in my mind, right?[940.407] [940.407][S05]And I’m trying to teach you, so you can’t directly see the reward -[943.343] [943.343][S05]you can only see like my feedback.[945.612] [945.612][S01] But my objective is to get you to say you like the sound that I’m making.[949.85] [949.85][S05] Exactly. [S01] Okay, let’s think of two sounds.[951.785] [951.785][S01]Let’s go for: Meep! And: Meeeeep![956.056] [956.056][S05] The first one.[958.292] [958.292][Hannah laughs] [S01] Okay.[960.594] [960.594][S01]Have you actually got something in your mind or are you just making -[962.763] [962.763][S01] This went on for a while. Beep Beep and [more noises here].[969.002] [969.002][S01]But with Jan’s feedback, we got Beep and Beep -[973.34] [973.34][S05] I’d go with the first one - [S01] Absolutely nowhere![976.677] [976.677][Jan laughs][977.778] [977.778][S05] This is really hard because of the expression problem actually.[981.748] [981.782][S01] I mean ultimately - but if I was an actual AI[984.351] [984.351][S01]I’d have I’d have gone through 10,000 iterations by now.[987.855] [987.855][S05] well there’s you have the, have a human loop[990.257] [990.257][S05]that reviews all of that so it is it is kind of slow um[994.394] [994.394][S05]but this is actually a problem that we have in our systems is that,[998.298] [998.298][S05]like in order to give useful feedback, you have to have useful examples, right?[1003.604] [1003.604][S05]In this case you produced sounds that are very similar,[1006.84] [1006.84][S05]and like the sound I had in mind[1008.008] [1008.008]was like very different [S01] Totally different[1009.476] [1009.476][S05] So I, I don't really like have the opportunity[1012.412] [1012.412][S05]to give you a very useful feedback, right?[1014.248] [1014.248][S01] Damn My optimization strategy was terrible here, okay [laughs][1017.251] [1017.251][S05] But this is like basically this is the same thing would come up[1021.088] [1021.088][S05]with a backflip right like if your robot just like lies on the floor[1024.291] [1024.291][S05]in like two different ways, like what are you going to do?[1026.36] [1026.36][S01] But you would hope though after I’d have maybe a hundred go’s at it,[1030.297] [1030.297][S01]I’d end up with something that began to approach what you had in mind.[1034.468] [1034.468][S05] yeah, definitely, definitely.[1036.703] [1036.703][S01] Is it like - oh no, I’m not allowed to ask any questions[1038.639] [1038.639][S01]am I, because I’m an AI.[1039.74] [1039.74][S05] I mean ideally this is at some point[1041.341] [1041.341][S05]this is what we want our assistants to be able to do, right?[1044.378] [1044.378][S05]Or just like describe the sound to you in actual language[1047.347] [1047.347][S05]and then you could just do it, that would be really cool, right?[1049.149] [1049.183][S01] Yeah.[1050.117] [1050.117][S05] So this is the kind of research that we want to do in the future,[1052.553] [1052.553][S05]that’s like the kind of systems that we want to figure out how to build.[1055.789] [1055.789][S01] Oh, okay, actually. Everything that I have done so far has been like voices.[1059.993] [1059.993][S01]You didn’t say that it had to be a vocal thing, did you?[1063.564] [1063.564][S05] Yeah - [S01] Okay, hang on.[1065.165] [1065.165][S01]Okay, alright, how about this, how about this.[1067.067] [1067.067][S01][clicks tongue]. And [whistles].[1070.637] [1070.637][S05] The first one. [S01] Ooh.[1071.972] [1071.972][S05] The first one is actually what I had in mind.[1074.107] [1074.107][S05][Laughs]. This is great.[1075.008] [1075.008][S01] I was so far away.[1077.744] [1080.147][S01]The only problem with setting up reinforcement[1082.382] [1082.382][S01]learning with a human standing over it offering feedback at every stage[1087.321] [1087.321][S01]is that it is monumentally slow.[1090.524] [1090.524][S01]Now it would take an agent months if not years to master a game like Atari.[1095.963] [1095.963][S01]And even then, you’d need to hire a pretty large group of poor students[1100.1] [1100.1][S01]to do the boring job of supplying feedback in real-time.[1104.438] [1104.438][S01]But there is an alternative.[1106.24] [1106.24][S01]You can rustle up a slightly more sophisticated learning partnership.[1111.612] [1112.212][S05] So what we do when we actually built these systems is[1115.749] [1115.749][S05]that we don’t literally do the experiment that we do now.[1118.852] [1118.852][S05]But instead we have - we train a neural network -[1122.389] [1122.389][S05]a second neural network that learns how I would give feedback as a human[1128.362] [1128.362][S05]and then the neural network can teach you[1130.531] [1130.531][S05]because it can just oversee all of the things that we are doing.[1133.333] [1133.333][S01] By now neural networks have become really good at spotting patterns.[1137.738] [1137.738][S01]Dog, not dog. Backflip, not backflip.[1141.341] [1141.341][S01]Perhaps you don’t always need a human in the loop laboriously giving feedback.[1146.513] [1146.513][S01]Why not have two agents - one attempting the task,[1150.284] [1150.284][S01]and the other deciding if it succeeded.[1153.62] [1153.62][S05] The reason why this works very well[1155.355] [1155.355][S05]is because evaluating the objective[1158.091] [1158.091][S05]is an easier task than producing the behavior that achieves it.[1163.197] [1163.197][S05]So you can have something like the for example[1165.432] [1165.432][S05]the and the back-flipping robot all the neural network has to do[1168.869] [1168.869][S05]is like look at what the robot is doing and see whether it’s a backflip.[1173.44] [1174.308][S01] You’re listening to DeepMind the podcast, a window into AI research.[1179.446] [1180.147][S01]But of course where this stuff really comes alive[1182.749] [1182.749][S01]is where you take the ideas - take them out of a computer simulation[1187.554] [1187.554][S01]and allow your imagination to roam into the world of embodied AI.[1193.16] [1193.16][S01]Robots that learn how to cook, robots that learn how to pack fruit in crates,[1198.465] [1198.465][S01]tuck you into bed and perform backflips.[1201.869] [1201.869][S01]It’s time to visit the DeepMind robotics laboratory.[1205.973] [1207.274][S03] we are standing right outside of the DeepMind robotics lab.[1210.978] [1210.978][S01] This is your lab? This is where you spend your days?[1213.447] [1213.447][S01]Well it’s not just my lab, but yes, this is our lab.[1215.482] [1215.482][S03] Well it’s not just my lab, but yes. It is our lab.[1216.917] [1216.917][S01] Can we go inside? [S03] Yes. We’ll get our badge . . .[1218.752] [1218.752][S01] This is Jackie Kay, a software engineer.[1222.055] [1222.055][S01]Packed to the rafters with robots.[1224.124] [1224.124][S03] Yes. [S01] It’s very cool![1226.56] [1226.56][S03] We’re basically running out of space.[1228.495] [1228.495][S01] It’s quite noisy in here.[1230.497] [1230.497][S03] Yeah, we have a lot going on today.[1232.332] [1232.332][S01] This place isn’t quite the high security basement laboratory[1235.869] [1235.869][S01]you’d imagine.[1237.204] [1237.204][S01]There are half assembled robot arms and machine parts[1240.841] [1240.841][S01]scattered all around the place. Some look like mechanical hands,[1244.745] [1244.745][S01]others quite a lot like red kitchen aide style food mixers.[1249.116] [1249.116][S01]And curiously, there is a SpongeBob Squarepants[1252.119] [1252.119][S01]mascot hanging from the ceiling.[1254.454] [1254.454][S07] Hey kids, check it out! [Jackie laughs][1256.657] [1256.657][S03] I think we use it as kind of a punishment of somebody[1259.96] [1259.96][S03]like leaves a tool lying around when they are supposed to put it back,[1264.198] [1264.198][S03]we’ll like put a SpongeBob in his space or something[1266.5] [1266.5][Jackie laughs] you know. [S01] That seems like a fair punishment.[1268.268] [1268.268][S01]Much of the work in this room focusses on getting robotics arm[1271.872] [1271.872][S01]to learn how to do simple tasks.[1275.042] [1275.042][S01]Each robot is bolted to the floor in its own cordoned off area,[1278.879] [1278.879][S01]and as we come in the door, there’s one that’s caught my eye.[1282.382] [1282.382][S01]You know that game that you play when you’re a kid um[1285.285] [1285.285][S01]and you’re you have a tennis bat and a ball attached to it[1289.356] [1289.356][S01]and then you kind of play tennis on your own,[1291.491] [1291.491][S01]it sort of looks like a robotic version of that.[1293.927] [1293.927][S03] It’s called the ball in a cup. [S01] Oh, so it actually is![1296.864] [1296.864][S03] Yes. It is exactly that.[1298.532] [1298.532]It is trying to swing the ball into that little basket. [S01] Oh![S03][1302.469] [1302.469][S03]So you can see it’s swinging the ball around a little,[1305.572] [1305.572][S03]and it’s actually checking position of the yellow ball um[1308.909] [1308.909][S03]and it’s trying to minimize the distance[1311.712] [1311.712][S03]from that ball to the area inside the basket, um -[1315.582] [1315.582][S01] Quick! it did it! [S03] Oh - it got it -Yes![1317.451] [1318.418][S01] Not exactly a smooth delivery of the ball into the cup -[1322.489] [1322.489][S01]more a slightly clumsy, lucky flip.[1325.759] [1325.759][S01]Of course if you’re ultimate goal was to make a perfect cup[1329.696] [1329.696][S01]and ball playing robot,[1331.431] [1331.431][S01]you could directly program a machine to do that without fail.[1335.669] [1335.669][S01]You can build robots that perform simple tasks in much more elegant ways.[1340.941] [1340.941][S01]But that’s not really the point.[1343.11] [1343.11][S01]Here’s Raia Hadsell, Senior Research Scientist at DeepMind.[1347.114] [1347.114][S02] A robot is a machine that can take over a task.[1351.885] [1351.885][S02]One can say that in the broadest sense[1353.253] [1353.253][S02]that your dishwasher at home and your vacuum cleaner are both robots[1356.557] [1356.557][S02]because they do something complex, they run on their own,[1359.193] [1359.193][S02]they have some autonomy to them of course[1361.295] [1361.295][S02]we also like to think about robots about having some intelligence[1364.898] [1364.898][S02]and then you get more into the realm of a robot with AI.[1368.502] [1368.502][S02]And I’m particularly interested in saying[1371.071] [1371.071][S02]how can we take this AI technology[1373.574] [1373.574][S02]and make it work on robots so that we have embodied AI.[1378.078] [1379.546][S01] And that is an important distinction.[1381.882] [1381.882][S01]The focus of the research in this robotics lab[1385.352] [1385.352][S01]isn’t to create robots that are told what to do,[1389.122] [1389.122][S01]but have them learn their own skills[1391.525] [1391.525][S01]much in the same way that other agents do here.[1394.494] [1394.494][S01]The robots here are what’s known as embodied AI.[1398.932] [1398.932][S03] So let’s say yeah, the task is a robot that’s trying to lift a box um[1405.372] [1405.372][S03]I as the programmer in the kind of traditional setting[1409.576] [1409.576][S03]would say go to box, open hand,[1413.881] [1413.881][S03]move hand um you know 30 centimeters towards the box, close hand.[1420.954] [1420.954][S01] But this is different.[1422.322] [1422.322][S03] Yeah, this is sort of taking what we want[1426.527] [1426.527][S03]and then figuring out how to accomplish what we want.[1429.363] [1429.363][S01] Every few seconds this particular robot[1431.164] [1431.164][S01]will have an attempt of getting at ball into the cup[1434.535] [1434.535][S01]before pausing, resetting, and having another go.[1438.605] [1438.605][S01]And every now and then, by chance, the ball lands in the cup[1443.01] [1443.01][S01]and the robot is rewarded with a positive score.[1445.879] [1445.879][S01]Just like if it was playing a computer game.[1448.882] [1448.882][S03] The reset in between the training episodes[1452.119] [1452.119][S03]where it untangles itself or flips the ball out of the cup,[1455.522] [1455.522][S03]those are scripted but then when it actually tries to accomplish the task,[1461.195] [1461.195][S03]that is a policy which it has taught itself through experience.[1464.431] [1464.431][S01] Over time from everything it learns from the schools it receives,[1468.635] [1468.635][S01]the robot builds a picture of how the ball moves,[1472.706] [1472.706][S01]and how this relates to the robot’s own movement.[1476.31] [1476.31][S01] So had we come in right at the very beginning bit, what would we see?[1480.714] [1480.714][S03] Just completely random noise,[1483.65] [1483.65][S03]probably very chaotic swinging around, ah - oh![1487.688] [1487.688][S01] Oh it did it in one! It did it in one![1489.489] [1489.489][S03] That was quite - that was wow! That was like in three seconds.[1493.627] [1493.627][S01] So will it know now that that movement gave it a successful result?[1499.266] [1499.299][S03] Yes.[1500.3] [1500.3][S01] And so the next time that we see this do we expect it[1502.936] [1502.936][S01]to be better than when we walked in?[1504.905] [1504.905][S03] Yeah, umm it will try similar actions[1508.175] [1508.175][S03]ah that gave it a positive reward or a success.[1512.145] [1512.145][S01] It’s just sort of sitting there quite smugly right now.[1514.047] [1514.047][S03] Yes.[1515.749] [1523.023][S01] There is a big advantage to getting the AI to figure out tasks for itself.[1528.095] [1528.095][S01]You’ll end up with a robot that is much more flexible out of the box.[1532.566] [1532.566][S01]It doesn’t matter what you want it to do - tie a knot,[1535.502] [1535.502][S01]stack some bricks, peel a banana.[1538.305] [1538.305][S01]Just as long as you can clearly communicate your objective,[1541.675] [1541.675][S01]you don’t need to specify a long list of instructions for these robots.[1546.346] [1546.346][S01]And perhaps they’ll come up with a new way of banana peeling[1549.149] [1549.149][S01]that you haven’t thought of.[1551.518] [1551.518][S01]But there is also a big drawback in training AI[1555.122] [1555.122][S01]that has a physical body - they are much slower to learn[1559.193] [1559.193][S01]than all of the disembodied agents you’ll find in this building.[1562.93] [1562.93][S01]The ones that only exist inside a computer.[1566.5] [1567.067][S03] Other researchers are able to take advantage of parallelism -[1570.904] [1570.904][S03]that is they run their environments in simulation, on computers,[1575.742] [1575.742][S03]and they can run them in parallel on hundreds of compute -[1579.479] [1579.479][S03]different computers.[1580.781] [1580.781][S03]We're all gathering data about this environment[1583.483] [1583.483][S03]they’re trying to learn something about.[1585.285] [1585.285][S03]We only have let’s see there’s 4 robots here[1589.122] [1589.122][S03]and they are frequently not running the same experiment,[1591.959] [1591.959][S03]so it might just be one robot collecting data.[1595.095] [1595.095][S03]And that means our training will be orders of magnitudes slower[1599.032] [1599.032][S03]for comparable tasks,[1600.501] [1600.501][S03]compared to ones that are running in simulation on computers.[1603.537] [1604.404][S01] Progress is slower in this room, and you can tell.[1608.809] [1608.809][S01]The agents here are a little less accomplished.[1613.247] [1613.247][S01]In another corner another robot is trying to pick up a Lego brick[1617.751] [1617.751][S01]with a hooked gripping device, kind of like a claw,[1621.021] [1621.021][S01]and there’s a rather ominous box of mangled Lego bricks next to it.[1625.692] [1625.692][S03] In the exploration phases of training[1627.895] [1627.895][S03]it will sort of randomly open and close its gripper,[1630.23] [1630.23][S03]and I think this one has some shaping to ah close its gripper[1634.535] [1634.535][S03]when it detects it’s close to the brick.[1636.57] [1636.57][S01] Ah - close![1638.639] [1638.639][S03] And then it also has a fixed training time [S01] ah![S03][1641.375] [1641.375][S03]so after some number of seconds, it just will give up[1644.845] [1644.845][S03]and go back to the start and then try again.[1648.482] [1648.482][S03]Lego is sort of this building block to general purpose manipulation.[1655.989] [1655.989][S03]If you know if we can stack you know two bricks together,[1658.959] [1658.959][S03]we can then do kind of arbitrarily[1660.794] [1660.794][S01] It’s going to break it![1662.329] [1662.329][Jackie laughs] [S03] Oh it’s fine.[1664.398] [1664.398][S01] sorry I got distracted by the robot smashing up the Lego.[1667.267] [1668.268][S01] There is real potential here, and so Jackie and the team[1672.539] [1672.539][S01]are constantly trying to find ways of speeding up that learning process.[1677.211] [1677.211][S03] One technique ah that we are looking into in order to[1681.248] [1681.248][S03]in that some of the researchers here[1683.183] [1683.183][S03]have done some really cool work on in the past is something we call[1687.02] [1687.02][S03]Sim to reel - or simulation to reality transfer[1691.859] [1691.859][S03]which is where you take a simulation on a computer that models your robot -[1698.365] [1698.365][S03]and we can learn kind of in broad strokes what the robot is like,[1702.336] [1702.336][S03]how its actions affects its environment[1704.404] [1704.404][S03]and how it can do something similar to a task it’s trying to learn in real life.[1710.577] [1710.577][S03]So once we figure out all of that in simulation,[1713.38] [1713.38][S03]without even touching the real robot,[1715.315] [1715.315][S03]we can transfer the data it’s collected onto real hardware.[1719.586] [1719.586][S01] So you’re - you can cheat basically.[1721.522] [1721.555][S03] Yeah.[1722.456] [1722.456][S01] You can cheat by imagining the real robot within a computer[1727.327] [1727.327][S01]calculating all the physics that would happen in real life[1730.13] [1730.13][S01]and then use the same techniques.[1731.765] [1731.765][S01]You used that army of computers to give you a bit of a head start[1735.002] [1735.002][S01]before you even apply it to the real physical robot.[1738.105] [1738.105][S03] Exactly.[1739.039] [1739.039][S01] So the real robots end up acting the same way as the simulated robots do.[1743.911] [1743.911][S03] Well they started out acting the same way[1746.079] [1746.079][S03]as the simulated robots do and then as they train more,[1750.384] [1750.384][S03]they might start behaving slightly differently or better[1753.153] [1753.153][S03]when they go into reality.[1754.988] [1757.324][S01] Using simulation might give you a head start on reality[1760.394] [1760.394][S01]but it’s never going to match precisely.[1762.729] [1762.729][S01]The real robot has to contend with grip, friction, gravity, wear and tear.[1768.602] [1768.602][S01]All of which play important roles in the real world,[1771.772] [1771.772][S01]but none of which will be perfectly represented inside the computer.[1776.476] [1776.476][S01]And all of that means, well -[1778.679] [1778.679][S01]I think science fiction may have set some false expectations.[1783.383] [1784.284][S01] One thing um that I am a little bit surprised about being in here[1789.022] [1789.022][S01]um don’t take this the wrong way, but these robots are a bit rubbish[1793.227] [1793.227][S03] Yes, it’s true, I mean, we’ve got a lot of work to do.[1797.931] [1797.931][S01] But as with so much of what happens here at[1800.1] [1800.1][S01]DeepMind, it’s not so much about these exact agents,[1803.604] [1803.604][S01]it’s not about cup and ball[1805.639] [1805.639][S01]or Lego stacking, it’s about the type of intelligence being acquired,[1810.677] [1810.677][S01]and how that fits into the bigger picture.[1813.38] [1814.114][S03] we want to demonstrate general purpose physical intelligence.[1818.018] [1818.018][S01] Physical intelligence? [S03] Right.[1819.786] [1819.786][S03]So contrasting that with kind of an intelligence that’s not embodied,[1825.225] [1825.225][S03]ah which maybe it can learn to play games,[1827.261] [1827.261][S03]or maybe even understand language, physical intelligence is looking[1832.399] [1832.399][S03]at how the physical actions of your body affect the real world[1836.937] [1836.937][S03]ah so we want to take a wide variety of tasks,[1840.04] [1840.04][S03]playing with objects, using tools,[1843.677] [1843.677][S03]ah maybe walking around or running in the future[1847.481] [1847.481][S03]and we want to show that robots can teach themselves[1851.485] [1851.485][S03]those tasks- how to do those tasks.[1853.72] [1853.72][S01] In the physical world?[1855.122] [1855.122][S03] In the physical world, yes.[1856.823] [1859.226][S01] But physical intelligence is of course[1861.762] [1861.762][S01]only one type of intelligence - one string to the robot’s bow,[1866.567] [1866.567][S01]and DeepMind as we have seen dares to dream big.[1870.938] [1872.206][S01]Here’s Murray Shanahan with the big finish.[1874.942] [1874.942][S06] The holy grail of AI research[1876.944] [1876.944][S06]is to build artificial general intelligence[1879.513] [1879.513][S06]so to build AI that is as good at doing an enormous variety of tasks[1885.752] [1885.752][S06]as we humans are,[1887.521] [1888.155][S06]so we’re not specialists in that kind of way,[1890.624] [1891.491][S06]you know a young adult human can learn to do a huge number of things.[1895.195] [1895.195][S06]You can learn to make food, you can learn to um make a company,[1900.033] [1900.033][S06]you can learn to build things, to fix things, you can do[1902.87] [1902.87][S06]so many things to have conversations, to rear children, so all of those things[1907.307] [1907.307][S06]- and we really want to be able to build AI[1909.71] [1909.71][S06]that has the same level of generality as that.[1912.679] [1915.249][S01] If you want to know more about robotics and technical AI safety,[1919.686] [1919.686][S01]then head over to the show notes where you can also explore the world of[1923.09] [1923.09][S01]AI research beyond DeepMind, and we’d welcome your feedback[1926.693] [1926.693][S01]or your questions on any aspects of artificial intelligence[1929.997] [1929.997][S01]that we’re covering in this series.[1931.999] [1931.999][S01]So if you want to join in the discussion or point us to stories or resources[1936.003] [1936.003][S01]that you think other listeners would find helpful, then please let us know.[1939.339] [1939.339][S01]You can message us on Twitter or you can email us -[1942.242] [1942.242][S01]podcasts@deepmind.com[1944.378]