Steve: Our guest today is John Schulman. He is a research scientist at OpenAI. He’s the co-lead for their reinforcement learning team, where he works on designing better reinforcement learning algorithms, and also designing better training environments that teach agents transferrable skills. Much of the work that he does uses games or virtual worlds as a test bed. John, welcome to Manifold.
John: Thanks for having me.
Steve: I sort of know you through one of your OpenAI co-workers, Sam Altman, and really wanted to have you on the show because Corey and I are both fascinated by what’s happening in AI, and OpenAI is one of the most interesting places where that work is going on. Let’s begin with your academic background, or your childhood, and how you got interested in AI in the first place.
John: Going back to my childhood, there was a TV show called BattleBots, where people would build these combat robots, which were really RC-controlled machines, and fight them against each other in a ring. That really captured my excitement, and I decided to set out and build one of these with a few of my friends. At least that was a good intro to a lot of engineering, both mechanical and electrical. So that didn’t go so far — at least I moved on to other things when I got into high school. But at that point I started to read a little bit about AI through, for example, Niels Nielsen’s textbook, and got interested in the topic there. At the same time, I read Ray Kurzweil’s book about The Singularity is Near. I found that pretty persuasive, and that had a big effect on my thinking going forward.
Steve: Of AI people, it seems like I often meet people who are older and maybe lived through the AI winter, and they’re very pessimistic about what can happen in our lifetimes, versus people like Kurzweil, who are super optimistic. So it’s interesting to meet younger people like you, who maybe cut your teeth already on those ideas like the singularity and things which were much more futuristic. What do you think is the breakup among, say, people that you studied with, in terms of whether they’re AI pessimists or AI optimists?
Corey: Before you answer that question, John, I want to hop in. My role here is as the audience ombudsperson, so can either you or Steve explain what the singularity is? I guess this will come out, but what’s Ray Kurzweil’s view? I know you said this had an effect on your thinking. Did it make you happy? Did it make you scared?
John: As for what the singularity is, this is just the idea that AI — mostly AI, possibly other technologies — will become more powerful in a way that’s self-reinforcing, and you’ll have some kind of runaway effect. In the AI case, the idea is that you get smarter and smarter machines, and then at a certain point they can improve themselves. So the rate of progress increases, and eventually things become incomprehensible. That’s why it’s called a singularity, because you can’t really predict anything about what’s going to happen after it.
Corey: And this is the plot of many disaster movies, when the computers take over.
Steve: It could be utopian or dystopian, depending on what they do to us once they take over. We’ll get into that later in the podcast, so let’s stick with John’s background a little bit. So you, I think, attended the same educational institutions that I did. You were an undergrad at Cal Tech and you did your PhD at Berkeley, is that right?
Steve: Were you in computer science the whole time?
John: I majored in physics at Cal Tech, so also like you.
Steve: Oh, wow. Okay. And then you were smarter than me, you switched to computer science. [laughs]
John: I was pretty interested in physics in high school, and starting college I dabbled in a little physics research early on in my undergrad career. Then at a certain point I realized I liked doing physics, but when I would go and read about what new scientific developments were occurring, I was more excited about developments in certain other fields, like neuroscience and AI. So I was kind of split between understanding the human mind and how to build artificial minds. I ended up actually applying to grad programs in neuroscience, and that brought me to Berkeley. I did some lab rotations in neuroscience and I was pretty sure I would go into that field, but for my last rotation I ended up working with Pieter Abbeel, who works on robotics and machine learning. That was just on a whim because I liked his work a lot, so I thought I would learn something new by working with him. Then I ended up getting really excited about the projects I worked on with him, and a few months later I ended up transferring into the computer science program and finishing up my PhD in AI.
Steve: Was there a lot of background that you had to make up because you hadn’t done a formal CS undergraduate degree?
John: Fortunately, not too much. I’d already been studying a decent amount of machine learning, because when I was getting into neuroscience I decided my niche would be to apply machine learning ideas there. I was also just independently interested in the topic, so I’d already gone and read some textbooks and done some projects involving machine learning. So yes, it wasn’t too hard to switch over.
Steve: Got it. So what brought you to OpenAI?
John: Around the time I was finishing up grad school, I was trying to decide what to do next, and I was thinking about maybe joining some existing research lab, or maybe joining or starting a start-up doing something like robotics. At that point, there was a group of people who were starting OpenAI and who were putting together the ideas and the initial group. I had a couple of connections to them, both through friends at Berkeley and also through initial meetings about AI safety. People were starting to think about AI safety and talk about it, and I’d gone to some initial meetings on this topic, so I knew some members of the initial group. Then at that point there was kind of a game where no one was sure exactly who’d be joining the initial organization, but there were a lot of conversations, and it ended up coalescing at a certain point. I met with Greg Brockman and Ilya Sutskever early on and then Sam Altman, and things became more and more concrete after that.
Corey: What is AI safety?
John: AI safety means just how to make sure AI systems do what we want, rather than have some unexpected and dangerous behavior. There are lots of ways you can optimize for something that seems reasonable and then get something you don’t want.
Steve: So for the audience, I’m going to say a bit about what I know about OpenAI, but I’m sure, since you were present at the creation, you can flesh this out much more. My understanding was that OpenAI started as a kind of reaction against DeepMind and Google, and some kind of almost fear and loathing that Demis Hassabis and company were going to solve the problem before the rest of the world could catch up. So the idea was to create a kind of open research institute which would share its results fairly freely with the public, and that the backing of some very prominent, and, perhaps wealthy people, I think, like Elon Musk, was involved in creating OpenAI. Is that accurate at all?
John: Yes. I don’t want to name specific names, but the reaction against DeepMind was not… That’s not what everyone’s motivation was. That was maybe one motivation [with] at least one person who was part of the initial effort, but that wasn’t the main motivation, and that became less of a driving force over time. I think now we view OpenAI as a pretty complimentary effort to DeepMind in terms of how it operates and how we go about things, but we don’t view them as rivals anymore.
Steve: Would you say though, that like DeepMind, you have as one of the core goals, getting to AGI, or artificial general intelligence?
John: Yes, definitely.
Steve: Because there are other AI research institutes at universities where that isn’t really one of their core focuses.
John: Right. Yes, I would say that’s definitely a similarity between OpenAI and DeepMind. We’re willing to say AGI and say that we’re working on it. We don’t have any shame in that, we’re excited about it. We don’t think it’s like crazy talk or anything.
Steve: Not only is it not crazy talk, you might even be willing to say, “AGI or bust”?
John: Pretty much, yes.
Corey: Can we again, for the audience’s sake, explain what AGI is and why it’s important, and why people think it might be crazy to go for it or dangerous to go for it?
Steve: Yes. Everyone now, I think, believes in narrow AI, which is machine learning focused on very well-defined problems, like playing chess or playing Go, or recognizing faces. In those contexts, it’s pretty clear that machines can achieve superhuman performance. The question is whether the kind of generalized learning and generalized reasoning that humans are capable of, will some day be instantiated in a silicon machine. That’s more what AGI is. And I think the community might be split between people who think AGI is a realistic possibility, say in our lifetimes or in the next hundred years, and other people who might think it’s going to take hundreds of years or never happen at all.
Corey: I think there are actually a couple of dimensions to the problem, because the problem is generalized along… it gets tasked, generalized along different axes. So you could have different types of pattern recognition tasks, and the question is, could you have a machine that could, say — take the language example — both answer questions and write paragraphs? That just might be thought to be a kind of statistical problem. Then there’s the further problem of actually reasoning, which many people think current systems don’t do very well, if at all. So the question is, AI’s presumably going after both of those axes across different types of tasks that might be classified as pattern recognition or statistical prediction, and also trying to develop systems that reason in a way that current systems don’t. Would that be accurate?
John: Yes, I agree with that. I would also say one of the conclusions of the past few years of the deep-learning revolution is that there’s not too much of a strict separation between pattern recognition and more sophisticated types of thinking and reasoning. So I would say it’s not clear. This is all kind of on a continuum. There are some tasks that involve very short, quick computations or reactions, and it’s easy to train neural networks to do those things. Some computations involve more steps of computation — we would call those reasoning — but it seems to be harder to train neural networks to do those kinds of things. But it’s not fundamentally different, at least so far it doesn’t seem to be.
Steve: I think you’ve touched on several topics, Corey, that I think we want to get into in more detail in the rest of the podcast. Before we leave the discussion of the organization OpenAI, I just wanted to ask John about the recent change in its funding model. So again, this is just my layman’s understanding, but they went from being a more classical not-for-profit organization to an organization which might be able to, in the future, create profits, and somehow the investors in it will, I guess, not take profits until it’s exceeded — I’m sorry, they will cap their profits at something like 1000x, and everything beyond that they will deny themselves and plow back into the entity. Is that correct, John?
John: Right. We started out as a non-profit, then we made a transition into an LP, limited partnership. It’s a capped profit structure, which means that investors get a certain maximal multiplier on their return, which is 100x for the initial pool of investors. So this didn’t actually change the mission or the day-to-day operation at all, it’s just a structure that made it possible to raise more money. As Sam Altman, the CEO, has said, investments in OpenAI are kind of in between an investment and a philanthropic donation, so this was a structure that makes it possible for people who are managing large amounts of money to make a big donation.
Steve: You say it’s a donation, but is it still tax-deductible under the new structure?
John: Probably not, but this is outside of my area of expertise.
John: I’m pretty sure it’s not.
Steve: Okay. I can think of two obvious reasons why you guys would need more money and bigger budgets. One is just raw compute, the ability to do really big sustained calculations. Then obviously the other one is the compensation that you AI guys make these days. To hire really top people, you have to pay a lot of money. Were those the main drivers for changing the financial structure?
John: Yes, that’s right.
Steve: Was there any pushback from the idealistic founding group that said, “Hey, but if we have to make profit, or we start orienting a little bit toward making profit, that it will change the level of openness that we’re able to have”?
John: There was definitely some debate about this topic, but eventually we found a version of this that we were all happy with.
Corey: So are you still committed to publishing all of your results or, in some sense, putting them in the public domain, or has that initial pledge been walked back a little bit?
John: Despite the name, we never pledged to publish everything. We tend to be open by default, meaning we’ll release our results and software if there’s no reason not to. It’s not always the right thing to do to release something, and that might be more true in the future if there’s more of a dual-use scenario where some method can be used in a dangerous way.
Corey: So the primary motivation for secrecy — I guess maybe that’s too hard a word — is that it could have negative consequences not to hold back something you might want to patent and therefore make money off of. Was that also a reason for keeping back a result?
John: We’re not planning to… We’ve never patented anything. We don’t have any immediate plans to do so. I couldn’t say definitively that we’ll never do this or that, or never try to license something and make money off of it, so that might happen in the future. That’s a possibility now. I don’t know if we’re going to end up doing that.
Steve: Back to the question of AGI: I’ve seen survey results, and these surveys are often taken at more kind of academic meetings, or meetings with both academics and industry people working in AI. A typical question would be something like, “How far are we from AGI?”, and the typical mean response would be like, “We’re 50 to 100 years from AGI.” Do you feel like the population of your colleagues at OpenAI is sort of a select sub-population from that broader group that would’ve answered the survey? In other words, are you guys collectively much more optimistic than the field in general?
John: I would say so. There’s still a bit of a distribution. We have a survey every once in a while where we ask people this question. I don’t know exactly what the median prediction is, but I’m pretty sure it’s shorter than 50 years.
Corey: So to give our audience an idea of what kind of progress has been made, can you give us a few examples of what you think have been major advances in AI, tasks you think are on the cutting edge, that can give people an idea of how far we’ve come, say in the past 5, 10, 20 years, and what you think the major problems are looking ahead that are on that road towards AGI?
John: Well, AI is a pretty big field. So I would have to pick—
Corey: Yes, just pick something that you’re really familiar with.
John: Well there’s certainly been a lot of progress in natural language processing. I mean, GPT was — generative pre-trained transformer it stands for, something that OpenAI released earlier this year — this was a neural net that was trained to predict texts. I don’t know if we’re going to get into this in more detail later, but in summary I’d say this was one of the big recent advances, and this system is able to generate realistic-sounding text and also be used for a lot of other natural language tasks, like answering questions, trivia questions.
Steve: If you guys want, we can go into a little more depth about GPT. Is that okay?
Corey: Great. Yes.
Steve: So let me give my, again, layperson’s summary that you should correct. My understanding is that this neural net was trained specifically to predict the (n+1)th word, given n words, in a well-formed human sentence. That seems like a really simple objective function, but training just using that objective function produced something that, when given a sample of text which set a kind of stylistic tone, it could then produce paragraphs of kind of novel text in the tone of that original kernel. And I don’t think it always does it flawlessly — humans have to maybe inspect the output and throw out some fraction of the output as just being garbage — but the ones that are not thrown out are, typically, shockingly good. The analogy I would use is, they remind me of internet fan fiction. So if you feed it a kernel talking about Frodo and Bilbo fighting the Orcs, it then generates several paragraphs which could have been written by kind of a bad Tolkien fan — not bad, but it’s just not Tolkien — but it’s something that clearly understands something about Middle Earth, understands that Elves fight Orcs, that Orcs have axes and swords. It seems to have a fair amount of common-sense knowledge about the world in order to construct those paragraphs, and it created the internal structure in itself that has that common-sense understanding through just optimizing this ‘predict the (n+1)th word’ objective function. Is that a fair description, John?
John: Yes that’s right, and I would say this is a very simple statistical technique, but then there’s not a clear divide between what this system can do and, say, creative writing. It would be hard to come up with some set of tasks that you can definitely do with GPT and some set of tasks that you can’t. It seems like if you keep making this language model better and better, it’s not clear what it won’t be able to do.
Corey: This was something that I noticed in the paper, that there was a lot of discussion of zero-shot learning, the idea that you train it on one task, say generating the (n+1)th word given a sequence of words, and then it could go on to actually perform other tasks like answer questions without actually having been trained at answering questions. I think it’s interesting because this approach seems like it’s answering some basic questions, or at least it’s answering questions given constraints that occur in linguistics. One of the basic mantras of linguistics is that there’s no explicit training. People learn language without being told this sentence is grammatical and this sentence isn’t, or any explicit instruction. In some sense, the system’s learning without any explicit instruction either, although it’s not using any of the principles linguists have postulated to account for this, but it’s solving the same problems. And this zero-shot learning seemed like it’s a pretty interesting take on the idea that humans do novel things. Was that part of the idea, do you think, behind trying to train it in this way?
John: Yes, definitely. I think it’s very surprising — to me at least, and to most of the people who have seen what it can do — that it worked so well, and that this zero-shot learning works so well. Just by learning how to predict the next element in a sequence, you learn how to really blend into the context you’ve seen.
Steve: I think — showing my age here, in going back to the early stuff that I’d read as a kid about AI — I always thought one of the classic problems that AGI was going to have to overcome is, how is your AI going to get a common-sense set of facts or relationships about structures in the world? I think there was even some project — I don’t know if it was an MIT project — but there was some project where people were literally coding in facts like trees have leaves, grass is green, and just creating some kind of structured database that had factual relationships about the world. That, I think, just ran out of steam and failed. You could have maybe predicted it wasn’t going to succeed. But here, you are doing something where the neural net has some capability, and the subjective function is good enough that, just by feeding it tons and tons of sentences about the world, it then — without your understanding of exactly how it’s doing it — it’s creating some internal structure that encodes that information. So in a way, what’s significant about it is not so much the parlor trick that you can create human-like text or paragraphs, but that you have actually somehow encoded in this set of connections between elements in the neural net, all of that information that’s present in all of the human literature that’s ever been written. So to me, that’s the most conceptionally interesting aspect of it.
Corey: It’s raising serious questions about our previous models and how the mind works. We assumed before that there was some store of information back there that was independent of our language skill. We had to build up the store of information, and the language skill would draw on that in expressing itself. This suggests that the two are much more closely intertwined — at least theoretically, much more closely intertwined, maybe even in how we do it, not just in principle, and that the AI system was not somehow [deemed] unusual.
Steve: Corey’s background is in philosophy of language and linguistics, and also neuroscience, and I have always felt that he’s a little bit more pessimistic than I am about AGI. So I’m curious — and answer honestly, Corey, no revisionist thinking — before the results of this GPT-2 paper were available to you, would you not have bet very strongly against the procedure that they went through working?
Corey: Yes, I would’ve said no way in hell actually, to be honest with you.
Steve: Yes. So it’s an event that caused you to update your priors.
Corey: Absolutely. Just to be honest, when I was coming up, I was at MIT in the mid ’80s in linguistics, and there was this general talk about how machine translation just would never happen and how it was just lunacy, and maybe if they listened to us at MIT and took a little linguistics class they might actually figure out how to get this thing to work, but as it is they’re going off and doing this stuff which is just destined to fail. It’s a complete falsification of that basic outlook, which I think, — looking back, of course — had very little evidence — it had a lot of hubris behind it, but very little evidence behind it. I was just recently reading a paper in Dutch, and I just simply… First of all, the OCR recognized the Dutch language and it gave me a little text version of the page. I simply copied the page, pasted it into Google Translate, and got a translation that allowed me to basically read this article without much difficulty. That would’ve been thought to be impossible 20, 30 years ago — and it’s not even close to predicting the next word, or writing in the style that is typical of the corpus. Just one question, John. Do you have any idea how large the training set was for this system?
John: I don’t remember off the top of my head. I think the paper on GPT-2 probably specifies it. If I had to take a guess, it’s somewhere in the tens of gigabytes.
Steve: I could be wrong, but I vaguely remember it was something like 3 or maybe 10 gigabytes, and I think largely just scraped off the internet or something.
John: Yes, that’s right. The larger model that was tested was something like 1.5 billion parameters. General heuristics is that you should have at least as much data as you have parameters, so I think that would mean you need to have billions… You’re predicting billions of tokens in the corpus.
Corey: This is a pretty exciting result for anyone who’s not inside the field. Is this a big part of what OpenAI is working on now, is language processing, language comprehension? First of all, I guess I should ask, was this a surprise to you guys that this worked?
John: Yes, definitely. It was a surprise to probably everyone who thought… Definitely it updated us on how powerful we thought our methods, or certain classes of methods, would be. So I certainly updated in favor of thinking that if you just train the right kind of recurrent network, such as a transformer, which is the type of model which was used for GPT, there’s hardly any limit to what kind of computation it can do, as long as you train it on the right kind of data. So it definitely updated me a lot, even though I’m not directly working on the language domain.
Corey: One other thing I thought was really exciting about this paper is that it was unsupervised. In most learning up until now, you basically have a set of data and you give it to the network, and you’d have a sense of what the right or wrong answer was. Here you just simply fed it a corpus of sentences, or actually words, and then the corpus itself will of course have the next word in it, so you could just cut it loose on a corpus and see whether it got the next word right, and do this continually without actually having to intervene. No human was intervening, basically telling the network, “You got that one right, you got that one wrong.” And that just seems like it opens up enormous possibilities for training networks, because human time is not infinite, but the web, pretty close to, is. So I guess there’s two questions implicit in here. Both the methods you use and the result, has that changed the trajectory of what OpenAI is maybe thinking about now as possible and practical?
John: Yes, definitely. It’s increased our love of scaling things up. That was already present, but the GPT-2 results reinforced it. Also, it just changed our outlook on what problems are solvable in the near future. As for the unsupervised learning, I think one way to look at it is, in supervised learning you’re only predicting part of the data, so you have an image and a label and you only predict label-given image. Whereas what GPT-2 does is, it predicts everything, it predicts every bit of information you’re presented with. Before seeing this type of result, you might think, “If I predict everything, then I’m going to waste all of my neural network’s capacity on predicting things that don’t matter.” That’s probably true to some extent, that you waste some capacity on predicting irrelevant details, but one lesson is that actually predicting everything is pretty good as long as you have a big enough network. If you have a big enough network, you can soak up all the information and predict the useful information as well as the useless stuff.
Steve: From an information-theoretic viewpoint, you can imagine there’s a much smaller corpus that’s been filtered for redundancy and other things that it could’ve trained on, and maybe gotten just as good with less compute and less, smaller corpus.
Corey: I think they actually did do a kind of curated network. I believe they used Reddit feeds that had a certain number of edits and a certain number of approvals, which is a clever way of getting the crowd to—
Corey: … vet your data set.
Steve: … ensuring it’s high-quality corpus.
Corey: Can you give us an idea of how big was the network? How many nodes?
John: The network has 1.5 billion parameters.
Corey: Parameters are nodes?
Corey: Are parameters nodes, in the sense of what we ordinarily think of as a network’s—
Steve: Maybe connections.
Corey: … connections?
John: Oh, so these are weights, yes. There are a bunch of weight matrices in any neural network, and most of the parameters are in weight matrices. So you’re just doing a bunch of linear operations, and then there’s also some nonlinearities thrown in. This project used a model called the Transformer, which was proposed a couple of years earlier. It’s a particular kind of neural network that makes a lot of sense on time series data — but not only that, now it’s being used in a lot of different places —but it’s pretty good for time series data, and it’s made up of lots of weight matrices and some nonlinearities thrown in like any other neural network. It’s something like 12 layers, so it’s a pretty deep network, though not one of the deepest networks ever trained.
Corey: For our listeners, can we perhaps draw an analogy between the human brain and this network? The weight matrices you’re talking about would be connections between what I think are conventionally called nodes in the network, so the analogy would be connections in the human brain. Could we say that it would be something on the order of 1.5 billion connections?
John: Yes, that’s right, 1.5 billion synapses would be the analogy. The human brain has something like 1015 synapses, so it’s still something like six orders of magnitude larger, five or six.
Steve: So from the more abstracted or theoretical viewpoint of neural nets, there’s this notion that you have this structure — we kind of stumbled upon it, or maybe we stumbled upon it because we knew something about how our own brains worked — but the idea is that you have something now which is expressive enough that it can encode a very large set of possible high-dimensional functions — so mapping some high-dimensional input to some other high-dimensional output — and it’s rich enough that it can capture a huge set of functions that we might be interested in, but the training of it is, in a sense, quasi convex. In the early days, like in the ’80s and ’90s when there were a lot of physicists actually going into neural net stuff, I remember everyone was afraid that, as you were trying to optimize the network, you could get caught in a local minimum and just not be able to get out of it. But now people understand better that that’s very unlikely, actually, to happen.
Corey: Can you explain what a local minimum is, and what it means to get caught in a local minimum?
Steve: Right. So if you have an objective function, there’s some thing, like goodness of guessing the (n+1)th word, given the nth word, and you want to optimize that function, you can think of it as an energy surface over the set of all possible values of the parameters. So in this high-dimensional space, there’s this energy surface. Think of a marble rolling down this mountain range on this energy surface. In the old days, people thought these things weren’t working well because, “Oh, the neural net got caught in the local minimum and it’s just trapped there, and it’s not able to find a really good set of weights that make it operate well.” And it turns out that wasn’t really the problem; the problem was they didn’t have enough data, or they didn’t have enough compute. And now, people have pretty good mathematical arguments suggesting that the chance of your getting caught locally is really quite small, in fact goes to zero as the dimensionality goes up. So you have the combination of a thing which is very expressive, so it can capture basically any set of really high-dimensional functions you want, given the right amount of training data; and secondly, the training of it is computationally feasible, because you’re basically just rolling a marble down a mountain but there are no valleys. If those two things are true, then the idea that “Oh, we’ll just go from 1.5 billion connections to… oh, let’s try a trillion connections, and let’s give it even more, larger corpus, we’ll just get something which is even more awesome” — there doesn’t seem to be any theoretical reason to expect that that little syllogism will be wrong.
Corey: In the past, in effect, you had little tricks, I guess, that they don’t do any more. Let me know if this is right, John. You’d inject noise into the network, kind of a random sequence of numbers, to bounce you out of what you thought were these local minima. Does that not happen anymore?
John: There’s already a lot of noise when you use stochastic gradient descent, which means you just choose random examples to train on. People have experimented a little bit with adding extra noise, but I don’t think the current verdict is that it really helps because there’s already enough noise from the SVD [singular value decomposition].
Steve: You’re showing your age, because people from our generation thought a lot about getting trapped in these little local valleys, and then there were these tricks of like “heating up the system” to try to bounce the marble out of that valley. Apparently in these modern contexts it’s really not an issue, so it’s kind of amazing. I think the positive scenario, optimistic scenario for AGI is just this, [that] there’s no obvious limitation that we’re going to run into. If we can get enough compute to do a trillion connections and we have enough training data, maybe unsupervised learning, we’ll be able to get even further, and the thing will be writing Shakespearean sonnets. I think it can already do that to some extent, but it’ll be writing incredibly beautiful things with a deep insight into the nature of the universe and the human condition.
Corey: Yes. We plan to get into this in a little while, but there’s two senses of optimistic which are possible. One is the technical possibility. To kind of out myself, one of my interests, John, partly in wanting to talk to you is, I’m kind of a critic of my old field of linguistics, and my sense is that linguistics hasn’t really progressed very far, and that in fact you can understand this by looking at the structure of the field, the fact that it’s heavily ideological, the fact that it was not very open to criticism, there are different camps, etc. etc. The other thought I have, and pose it to you, is my sense is that theoretical linguistics has contributed very, very little to the study of language in artificial intelligence. In fact, it seems like the research that’s gone on GPT-2, and other models, pays no attention to it at all. I think this raises really deep questions about the importance of linguistics or whether it’s telling you anything about the world. Now linguists argue, “Well, what these systems are showing is you can do language processing in a certain way, but that may not be how humans are doing it. So we’re, in linguistics, studying how humans do it. You guys are just studying how it’s possible to do it using some other type of system.” But I actually think these systems are raising much deeper questions about what linguists are doing, because I think there’s very little reason to think that their techniques are actually designed to focus on people, and they just may be models that are just simply inaccurate — maybe common-sense models, or at least closer to common sense. Anyway, that’s my little tirade against linguistics. I’d like to hear your thoughts.
John: Yes, I think a lot of fields actually had a similar kind of wake-up call when the methods they’d developed became obsolete by kind of dumber statistical methods. Something similar happened in computer vision, where people had developed all sorts of ideas based on geometry and understanding the visual system, and they were all blown away in performance by the deep neural nets that don’t really incorporate any of this world knowledge. So I would say that certainly, if the goal is to create conversational systems, or if the goal is to do AI, then you’d probably want to throw out most of the domain knowledge and just use systems that are trained on data. At least that’s the way things seem to be going, in the long run at least. I don’t know what the other applications of linguistics are, where it might be good to understand the structure.
Steve: You know, I think you could always say linguistics is still an interesting thing, because it’s describing human language and how we do certain things, but maybe it isn’t actually how our brain does it.
Corey: But that’s what Chomsky was denying for 50 years. His claim was there’s a kind of superficial level of description, but in fact linguistics is going after explanation.
Steve: Yes, okay.
Corey: It’s the explanatory level. It’s accounting for actually how the mind operates at a deeper level. That was the claim, constantly. It was reiterated in almost religious fashion, but the very fact that it was reiterated in a religious fashion often told you that there actually probably wasn’t much data to back that up, and there hasn’t been a lot of neuroscientific support for it. I think these systems raise the question of whether human minds are operating similarly or just some very large statistical machine without a lot of built-in structure. So this network — again, this network had no built-in structure of a domain-specific sort. Is that right?
Steve: Certainly not domain-specific. It had 12 layers, that was fixed, and then some structure of connections between the layers. Do you mind if we move— Oh, go ahead, John.
John: Yes, actually on the previous point, humans still have to come up with language at some point, so people spontaneously invent language, right? We still haven’t shown that neural networks like AI systems can invent sophisticated language to speak to each other with, so, I think there’s still a question there.
Corey: Well, I heard — and maybe this is a rumor — that at some point Facebook was experimenting with a kind of translation system and realized that the system had developed some kind of internal language for communicating between different parts of the network. And at that point in time, when they realized that they shut the program down.
John: Yes, that was a bit of a click-bait kind of article.
Corey: Oh, okay. All right.
Steve: That was the BuzzFeed version.
John: People are definitely working on spontaneous emergence of language, so trying to build some kind of system with multiple agents that have to accomplish goals together. The hope is that they’ll invent some language to communicate with each other. That’s an active area of research, but it’s still pretty preliminary.
Steve: Let’s segue in that direction because, John, I think you actually work on some of these agents in video game worlds, trying to accomplish goals, those kinds of problems. Is that true? There, you can imagine — I think I’ve even read of situations where they learn to communicate with each other, like if they’re playing Capture the Flag maybe they can signal to each other that one’s going to guard this flag and the other one should go after the other flag or something — in those contexts, where the agents are different autonomous, separate autonomous entities, you can imagine they have to invent some level of communication to win the game. But maybe we could just back up and tell us a little bit about your work in particular, the utility of these virtual game environments, and what the most impressive achievements are in that area.
John: I’m interested in reinforcement learning, which is learning by trial and error. It’s how you get an agent that can interact with the world and achieve its goals. That’s the general area, but the key thing is that, unlike, say, supervised learning, where there’s an input and there’s a correct output that you’re supposed to produce, in reinforcement learning you never get to see what the correct output is. You just get some kind of reward function that tells you if you succeeded or not. So I’m interested in improving our algorithms for reinforcement learning and, in particular, making them more sample efficient, which means allowing the agents to learn faster how to solve problems. I would say that’s one of the big weaknesses with our existing AI systems. They can eventually learn how to do complicated tasks, but it often takes a lot of experience, and experience in the real world is expensive. If you go and deploy some robot in the real world and it falls down a hundred times before it learns how to walk, it’ll probably break by the time it knows how to walk. So that’s not going to be any good.
Steve: In the virtual environments, I guess you guys have worked specifically on Dota, is that right? And DeepMind worked on… What were they working on, was it—
Steve: StarCraft, right.
Corey: So what is Dota? StarCrafts were non…
Steve: Maybe just explain Dota, what you guys worked on.
John: Yes, so Dota is one of these popular video games. It’s a team-versus-team kind of game, where there are five heroes on each team, and they can cast all sorts of spells on each other. You’re trying to make it into the enemy base and destroy their base. It’s a kind of fantasy game that’s played by millions of people.
Steve: So you have separate neural nets controlling each of the five members of your team? They don’t have any other way of communicating other than their actions in the game world, is that true?
John: Actually, the way we set it up, there’s a single neural network controlling all five of them.
Steve: Oh, I see.
John: So they don’t need to communicate, unlike humans.
Steve: Okay. That might be considered cheating, but of course it’s still an interesting problem to solve, right? So, five coordinated people.
John: I think the problem where you have five separate agents that don’t get to see each other’s screens would be a bit more challenging. Any tasks that you try to use AI on, you’re doing some amount of cheating just to make the computer able to interact with the task, as opposed to having — I don’t know, you can have a robot pressing the keyboard, that would be the most pure version of it — but everything involves some amount of cheating.
Steve: Sure. So in this case again, there’s no specific optimization for the structure of the game world? Do you start from zero or random actions and then you learn from that? Is that how it works?
John: We start with random actions. There’s some domain knowledge involved in defining like, how do you feed the game state into the neural network? So there’s some engineering there, but we tried to do the minimal amount of handcrafting, because often when you try to engineer a lot of stuff by hand you end up making your system less powerful, because you remove its ability to learn the optimal way of doing something.
Corey: So do you have a camera focused on the screen, and the neural net’s getting output from the camera? How do you actually feed in the game state?
John: There’s an API that tells you, “Here are all the friendly and enemy units that are visible, and here are their positions,” so we have a neural net that takes in the whole game state.
Steve: And you’ve reached a professional level, or even beyond professional level of play in Dota?
John: Yes. The bot was trained to the professional level and, in fact, to beat the world champion in a live match. So yes, in a way it’s superhuman. Then, after playing against the world champions in this exhibition match, we opened it up so that anyone could sign up and play it. So thousands of players around the world got to play against it, and it turns out people discovered some exploits against the bot. They didn’t discover it immediately, but certain very skilled and tenacious players were able to discover weaknesses and figure out how to amplify them so they could win a match.
Steve: I was actually told that in the very first victory of, I think it was called AlphaGo at the time, that instance of the neural net really had a bunch of weaknesses, and had the Korean champion played in certain directions, it could’ve destroyed it. So that’s why there was a long hiatus between that championship game and then the following set, because they needed to fix all of these weaknesses that could’ve been exploited. So I think it’s maybe not surprising that those could exist.
John: Yes. I think if you have one of these systems, it’s quite likely to have weaknesses. It’s never been trained to play against humans. It’s playing against itself. Hopefully it’ll discover most of the possibilities, but often there’s some very long chain of actions that you need to find a certain exploit, so it’s quite likely that there will be a weakness. I feel like now one of the limitations of both the OpenAI Dota system and AlphaGo is that, if someone does discover a weakness, the bot can’t immediately adapt to it. So ideally we would release this bot to play lots of people on the internet and someone discovers the exploit, but then the bot learns from that one match how to defend against it. Right now, the bot had to be trained on thousands of hours of experience, or maybe hundreds of thousands of hours, and it’s impossible to do a big enough update from that one game to fix the problem. So I think that’s where the sample efficiency problem comes in. You want to be able to learn from a small amount of experience.
Steve: Right. A human might realize, “Oh, there’s this weakness in my axe, and if I pulled it in the wrong way the other guy can smash it” or something, and then just stops doing that, but the AI can’t currently do that.
Corey: Isn’t this a kind of example of where “reasoning” might kick in? You actually would deliberate a little bit about what went wrong in this match, and call on background knowledge.
Steve: Right, and it currently doesn’t have anything like that capability.
Steve: I’m curious: if third parties are watching humans play against your bot, can they tell which one is the human and which one is the bot playing?
John: I don’t know actually. I’m guessing that a really skilled observer could pick out the play style of the bot and figure out which one is the bot, but I’m not sure.
Steve: But casual observers wouldn’t notice anything really strange.
Corey: Do you have any plans to try the experiment Steve described where you have individual bots controlling individual players, and then see if they can develop a method of communicating amongst each other?
John: Not that I know of, but I think that would be an interesting challenge for the future — maybe not with the game of Dota, but that’s definitely something we’re interested in pushing on in the future — I mean multi-agent environments, where the agents have to communicate with each other.
Steve: Could you talk a little bit about… In order to get beyond where we are now, are there any deep architectural changes that people are thinking about, like… I think I remember some different architecture with like a column that can transmit information across many layers or something. What’s on the horizon in that direction?
John: I’m sure there are going to be a lot more architectural innovations. The transformer, which is what we used for GPT, was only proposed a couple of years ago, so occasionally there are still these big innovations, and I expect to see some more of them. I have no idea where they’re going to come from or what people are going to do next. I think one interesting line of work that’s going on now is trying to understand the scaling properties of neural nets. What happens when you make them wider or deeper, and what really matters to determine performance? Is it just number of parameters or something else? What’s the optimal way to scale up a neural network? Also, just having some theory of what makes a neural network optimizable: like, if you give me a neural net structure that you propose, I should be able to tell if it’s any good without even running it, if you’ve got a good theory. So I think there’s a lot of progress on the scientific side, and I think that’s going to also help to drive some of the new innovations.
Corey: The column architecture you describe, Steve, is actually also inspired by cortical columns. I think one of the interesting questions is whether, going forward, you find that these inspirations from neuroscience are useful or not, or whether they’re just more or less what we’re familiar with. I think there are some theories about allowing more efficient processing of certain types of specific features perhaps.
John: Yes, I personally don’t think that neuroscience has provided that much inspiration on what architectures to build. Even the convolutional network, which is usually cited as something inspired by neuralness, by real brains and their structure, like the visual cortex, I think that’s kind of a stretch, because sharing weights across different parts of the visual field is impossible to do in biology, but that’s what convolutions are based on. I think it’s more that’s a natural way to do processing of images where you have some translational invariance. So yes, I don’t think neuroscience has told us much about how to design good architectures, but I think it provides more subtle forms of inspiration that the brain… Like, for example, we know that we can use introspection to get some insight into how we do certain types of processing. Certain processing involves listening to your mental monologue for a while and coming up with some plan, so I think we can get some inspiration about what our neural nets should be able to do and what we should be trying to build. But I don’t think we get much in terms of specific architecture ideas.
Steve: Yes, the introspection thing sounds almost more like we know a little bit about how our own thinking works and so that could inspire some strategy with neural nets, but it isn’t necessarily a result of any kind of fine mapping or imaging of the actual neurons in our brain.
John: Yes, right. I think it’s really hard to learn anything by using systems of neuroscience or by using any kind of brain mapping, because we don’t know that much about what algorithms the brain actually implements. We know maybe some of the very low-level details of how spiking neurons work and how development works and so forth, but there are a lot of levels of description between that and understanding what’s going on at the algorithmic level. So I think understanding the algorithms implemented by the brain is going to tend to lag really far behind development of algorithms in AI.
Steve: We may end up relying on the superhuman AGI to figure out how our brain works. It may be beyond human neuroscientists to figure out.
Corey: People have thought that for quite a while, actually. Our brain may be too complicated for us to grasp.
Steve: Right. I’m a little worried about that. [laughs]
Corey: Can we get to scary scenario part of our discussion?
Steve: Okay. Actually, I was going to ask one more question before we get to that. Are there any things that we’re capable of, whether it’s introspection, or consciousness, or the way our memory works, that it’s clearly not within the current architectures people use in silicon neural nets to achieve that? Are there any obvious gaps that you can point to?
John: I think there are certain abilities that humans have where it’s not totally obvious to me how to make an AI system do the same thing. For example, where you can learn from some experience that’s extended over time and after a single trial. Let’s say you leave some kind of glass on the edge of the table, then later it gets knocked off and breaks, and you conclude, “Oh, I shouldn’t have left that glass on the edge of the table.” You had to look at one sequence of time and compress it down into some short description, then you had to consider some counter-factual situation like, “If I hadn’t left it on the edge, what would’ve happened?” and then learn from that whole little experience. I just don’t see how our existing frameworks are going to do that, but who knows? A lot of problems that seem impossible kind of dissolve and end up being solved by some kind of simple idea. So we’ll see if that happens.
Steve: Yes, I would say it’s clear that human babies or children learn kind of folk physics models, and those are highly-compressed models where they realize the objects have some permanence, and they’re separate from other objects, and then they obey some Newtonian laws, things like this. Humans kind of figure these out just by going through their daily life, and it’s hard for me to imagine how you would get an artificial neural net to do that. But again, it could just be like the magic thing with GPT-2, where it learned a bunch of common-sense things about the universe just by reading a bunch of web pages. So who knows?
Corey: Well, I think at least the current theory is that our brains probably do have some innate structure built over the course of evolution that predisposes us to pick these things up pretty quickly. So you may have to build in some kind of structure in the neural net to allow it to carry out that kind of one-task learning. It would be quite a lot to ask an unstructured network to get it.
Steve: Right. So in that sense, maybe there is something to be learned from deep neuroscience… Getting to the dystopian scenarios, I had a specific set of sentences I wanted you to react to which is in that direction. Let me just read this and then you can tell me what you think. “It seems to me humans are just smart enough to create an AGI, so maybe in the next 50, 100 years we’ll make an AGI, but perhaps we’re not smart enough to create a safe one. So maybe AI safety is beyond our reach. Perhaps we should make humans smarter first, so there’s a better chance that our successors will look fondly upon us. In other words, genetically-engineered, super-genius humans might have a better chance at implementing Asimov’s laws of robotics.” What do you think about that?
Corey: Hold it Steve again. What are Asimov’s laws of robotics?
Steve: Oh, I can’t remember exactly what they are, but they’re basically like, “Robots should not harm humans.” I think there are three laws, and they’re very cleverly structured. They’re logically pretty complete, and the whole idea is that I think if robots obeyed those laws, you would go a long way toward fully implementing AI safety, although perhaps not 100% of the way.
John: I would agree with your sentence to the extent that smarter people are more likely to succeed at their goals. So it’s probably true that, if humans were smarter, we’d be more likely to solve safe AGI. I hope that it’s within reach of current human intelligence, and I think it probably is, but [it’s] hard to be sure. I think both of these fields are really exciting, both, genetics and AI, and both of them are going to be happening in parallel, and there might be some interaction between them, of course. It’s hard to predict how these things will play out.
Corey: I guess there’s a third field, of course, which is robotics, and when you take systems like you’re developing and put them inside a Boston Dynamics type robot, I think reasonably that most people would find that frightening, actually, because these robots are pretty adept at moving around terrain. They’re going to get faster and faster, and if you put weapons in their hands, it’s not all that much of a leap to see them killing humans.
Steve: I think the psychological dimension of having it in a kind of Boston Dynamics thing which ambulates around, I think that’s very strong. But I think actually in a practical sense, just drones — like a big predator drone with missiles, or a little drone that has a C-4 charge on the tip of it — those things already would be even easier to interface with an AI than these Boston Robotics things. And although they’re not as psychologically scary to us, they’re probably actually more effective as weapons.
John: It’s hard to scale things up in the physical world. So that makes robotics really hard, because you have to build robots and you have to deal with all sorts of complexities of being in the physical environment. But that also makes me less worried about physical robots going around and hurting people. I would be more worried about robots that live on the internet and can go and hack into systems, because they can instantly replicate themselves arbitrarily, many times.
Steve: The one that seemed fairly feasible to me was this idea that you could have relatively small drones — the battery issue is a problem, but maybe they could run on gasoline or something — but relatively small drones with very good face recognition and a little bit of explosive C-4 or something, and, basically you can assassinate whoever you want. You have these swarms of things flying around looking for Corey Washington, and when they see Corey they all dive into Corey Washington and blow him up, [laughs] or assassinate a president or something. That seems very close technologically to where we are now.
Corey: It seems like that’s actually much more frighteningly near to a possibility than AGI.
Steve: Yes, right. This just requires face recognition and just slightly better drones.
John: I’m not sure if there are any obstacles to that. You could also imagine that you can build defenses against that kind of thing, so you can have some kind of super-fast laser turret that’s going to shoot them down. I guess there’s a question of whether offense is easier than defense or not, and whether there’s some asymmetry there.
Corey: I think there’s probably no doubt that defense is very difficult, if you’re talking about an individual person protecting themselves against this.
Steve: One of the most entertaining things I’ve seen on YouTube in the last few years is something called First Person View Drone Racing. You’re basically wearing VR gear like this and you’re controlling a drone, and you see the view of the drone. These guys race these drones through really intricate courses, and the average speed is, I don’t know, 60, 100 miles an hour for these drones, and they have exquisite control. They’re flying them indoors, even in warehouses and stuff. It just gives you a sense of what’s possible, and how hard it would be to defeat, say, five of those drones coming after you. No way could a bunch of guys with guns shoot them down, I think.
Corey: So I think we have a sense that there are things that are probably much more frightening near term than AGI. What are the things that keep you concerned, John? Or does any of this make you—
Steve: He’s an optimist. [laughs]
Corey: Are you a technological optimist, or are you a kind of “the world’s going to be okay as a result of this technology” optimist?
John: I’m probably more of a technological optimist than most people, in that I— Do you mean, do I think it’s going to happen soon, or do I think it’s going to be good, by optimist?
Corey: Yes, there’s actually two senses of optimist, right?
Corey: If it’s going to happen soon, that’s one form, and is it going to be good for us?
John: Right. So I think I’m probably more on the soon end than most people. I’m not the most extreme person in that. As for whether it’s going to be good, I think so, I think it can be good. I’m maybe less worried about nightmare scenarios than most people. I’m more worried about what’s going to be the meaning of human life once we have AIs that are smarter than us. What does the future look like, and are we going to be happy with it? So even if we succeed at building AIs that are superhuman and they also value humanity and value us, are we going to be happy with this? What should the world look like?
Steve: Are you familiar with a science fiction writer called Ted Chiang?
John: Yes, I love his work.
Steve: I think he has a couple of short stories about what it’s like for humans after we’re eclipsed by the AGIs and their scientific frontiers have moved way beyond ours, and most of what we do is try to reverse-engineer the crumbs that fall out of their laboratories and things like that. So it could be very dispiriting.
John: Yes. That’s what I think about more than other issues.
Steve: We get in a time machine, we come out 10 years from now: What’s the most exciting thing that’s happened?
John: Let’s see. What I’m most excited about is having reinforcement learning systems that can learn really fast. For example, you present it with a video game it’s never been trained on before, and it can master it as quickly as a human. Obviously this is a ‘toy’ problem, because we’re still talking about video games which aren’t useful in the real world, but I think if we had that capability, that would enable lots of interesting real-world applications. So that’s what I’m personally most excited about.
Corey: Of those real-world applications that are possible, which ones do you think we might see in 10 years?
John: Something like AI personal assistants that are really good. I can imagine that being a big one. I think the world is getting more and more complicated. As we’ve gotten tools to deal with all of the complications, there are just so many different things you have to worry about. I think having AI to help manage all of these issues will improve human life a lot. So that’s one of the things I look forward to.
Steve: I keep getting invited by some San Francisco startup that’s building a very good — they claim very good — AI assistant that, if I join the beta, it will basically help me run my life, but I’m afraid to give it, and them, all of my personal details. [laughs]
Corey: These are assistants that are scheduling for you?
Steve: Yes, you can say to it, “I’m going to Boston next Thursday. Can you find me a good flight that arrives at night?” — that’s the kind of example that, I’m sure, that’s aspirational for them — or “Where’s the nearest good coffee place?”… you know, that kind of stuff. It’s Google Assistant but just at a higher level, and it has more direct access to your contacts list and very specific things about you, although maybe Google has that too.
Corey: That’s already fairly non-benign. [laughter]
John: Yes, thanks for having me on the show. It’s been a pleasure.
Steve: All right. Well John, it’s been a pleasure. We really enjoyed chatting with you and hope to have you back on the show some time.
Steve: All right, take care.
John: You too. Bye.
Corey: Thank you, John.