00:00:02.485 --> 00:00:04.080 [MUSIC] ELIZA STRICKLAND: Welcome   00:00:04.080 --> 00:00:07.600 to the Microsoft Research Podcast, where  Microsoft's leading researchers bring you to   00:00:07.600 --> 00:00:12.360 the cutting edge. This series of conversations  showcases the technical advances being pursued   00:00:12.360 --> 00:00:16.560 at Microsoft through the insights and  experiences of the people driving them.  00:00:16.560 --> 00:00:21.240 I'm Eliza Strickland, a senior editor at IEEE  Spectrum and your guest host for a special   00:00:21.240 --> 00:00:25.560 edition of the podcast. [MUSIC FADES]  00:00:25.560 --> 00:00:30.040 Joining me today in the Microsoft Booth  at the 38th annual Conference on Neural   00:00:30.040 --> 00:00:35.680 Information Processing Systems, or NeurIPS, is  Chris Bishop. Chris is a Microsoft technical   00:00:35.680 --> 00:00:40.120 fellow and the director of Microsoft  Research AI for Science. Chris is with   00:00:40.120 --> 00:00:44.400 me for one of our two on-site conversations  that we’re having here at the conference.  00:00:44.400 --> 00:00:46.240 Chris, welcome to the podcast. CHRIS BISHOP: Thanks,   00:00:46.240 --> 00:00:49.200 Eliza. Really great to join you. STRICKLAND: How did your long   00:00:49.200 --> 00:00:53.360 career in machine learning lead you to this  focus on AI for Science, and were there any   00:00:53.360 --> 00:00:56.800 pivotal moments when you started to think that,  hey, this deep learning thing, it's going to   00:00:56.800 --> 00:01:00.040 change the way scientific discovery happens? BISHOP: Oh, that's such a great question.   00:01:00.040 --> 00:01:04.560 I think this is like my career coming full circle,  really. I started out studying physics at Oxford,   00:01:04.560 --> 00:01:10.280 and then I did a PhD in quantum field theory. And  then I moved into the fusion program. I wanted to   00:01:10.280 --> 00:01:15.040 do something of practical value, [LAUGHTER] so I  worked on nuclear fusion for about seven or eight   00:01:15.040 --> 00:01:19.360 years doing theoretical physics, and then that  was about the time that Geoff Hinton published   00:01:19.360 --> 00:01:23.800 his backprop paper. And it really caught  my imagination as an exciting approach to   00:01:23.800 --> 00:01:29.480 artificial intelligence that might actually yield  some progress. So that was, kind of, 35 years ago,   00:01:29.480 --> 00:01:33.600 and I moved into the field of machine learning.  And, actually, the way I made that transition   00:01:33.600 --> 00:01:40.160 was by applying neural networks to fusion. I  was working at the JET experiment, which was   00:01:40.160 --> 00:01:45.880 the world's largest fusion experiment. It was  sort of big data in its day. And so I had to,   00:01:45.880 --> 00:01:47.512 first of all, teach myself to program. STRICKLAND: [LAUGHS] Right.  00:01:47.512 --> 00:01:51.400 BISHOP: I was a pencil-and-paper theoretician  up to that point. Persuade my boss to buy me   00:01:51.400 --> 00:01:55.640 a workstation and then started to play with  these neural nets. So right from the get-go,   00:01:55.640 --> 00:02:01.640 I was applying machine learning 35 years ago  to data from science experiments. And that was   00:02:01.640 --> 00:02:06.760 a great on-ramp for me. And then, eventually,  I just got so distracted, I decided I wanted   00:02:06.760 --> 00:02:11.480 to build my career in machine learning. Spent a  few years as a research professor and then joined   00:02:11.480 --> 00:02:17.800 Microsoft 27 years ago, when Microsoft opened its  first research lab outside the US in Cambridge,   00:02:17.800 --> 00:02:23.440 UK, and have been there very happily ever  since. Went on to become lab director. But   00:02:23.440 --> 00:02:30.200 about three or four years ago, I realized  that not only was deep learning transforming   00:02:30.200 --> 00:02:34.640 so many different things, but I felt it was  especially relevant to scientific discovery.   00:02:34.640 --> 00:02:40.960 And so I had an opportunity to pitch to our chief  technology officer to go start a new team. And he   00:02:40.960 --> 00:02:46.000 was very excited by this. So just over two and a  half years ago now, we set up Microsoft Research   00:02:46.000 --> 00:02:51.100 AI for Science, and it's a global team, and  it, sort of, does what it says on the tin.  00:02:51.100 --> 00:02:56.320 STRICKLAND: So you’ve said that AI could usher  in a fifth paradigm of scientific discovery,   00:02:56.320 --> 00:03:00.160 which builds upon the ideas of Turing  Award–winner Jim Gray, who described   00:03:00.160 --> 00:03:05.360 four stages in the evolution of science. Can you  briefly explain the four prior paradigms and then   00:03:05.360 --> 00:03:09.480 tell us about what makes this stage different? BISHOP: Yeah, sure. So it was a nice insight   00:03:09.480 --> 00:03:14.080 by Jim. He said, well, of course, the first  paradigm of scientific discovery was really   00:03:14.080 --> 00:03:18.800 the empirical one. I tend to think of some  cave dweller picking up a big rock and a small   00:03:18.800 --> 00:03:22.660 rock and letting go of them at the same time and  thinking the big rock will hit the ground first …  00:03:22.660 --> 00:03:24.280 STRICKLAND: [LAUGHS] Right … BISHOP: … discovering they land together.   00:03:24.280 --> 00:03:27.960 And this is interesting. They've discovered  a, sort of, pattern irregularity in nature,   00:03:27.960 --> 00:03:32.640 and even today, the first paradigm is in  a sense the prime paradigm. It’s the most   00:03:32.640 --> 00:03:37.760 important one because at the end of the day, it's  experimental results that determine the truth,   00:03:37.760 --> 00:03:42.920 if you like. So that's the first paradigm. And  it continues to be of critical importance today.   00:03:42.920 --> 00:03:48.520 And then the second paradigm really emerged  in the 17th century. When Newton discovered   00:03:48.520 --> 00:03:53.880 the laws of motion and the law of gravity, and  not only did he discover the equations but this,   00:03:53.880 --> 00:03:57.800 sort of, remarkable fact that nature  can even be described by equations,   00:03:57.800 --> 00:04:01.920 right. It's not obvious that this would be true,  but it turns out that, you know, the world around   00:04:01.920 --> 00:04:06.320 us can be described by very simple equations  that you can write on a T-shirt. And so in the   00:04:06.320 --> 00:04:10.440 19th century, James Clerk Maxwell discovered  some simple equations that describe the whole   00:04:10.440 --> 00:04:14.640 of electricity and magnetism, electromagnetic  waves, and so on. And then very importantly,   00:04:14.640 --> 00:04:19.080 the beginning of the 20th century, we had this  remarkable breakthrough in quantum physics. So   00:04:19.080 --> 00:04:23.160 again down at the molecular—the atomic—level,  the world is described with exquisite precision   00:04:23.160 --> 00:04:29.320 by Schrödinger's equation. And so this was the  second paradigm, the theoretical. That the world   00:04:29.320 --> 00:04:35.840 is described with incredible precision of a huge  range of length and time by very simple equations.  00:04:35.840 --> 00:04:41.800 But of course, there's a catch, which is those  equations are very hard to solve. And so the   00:04:41.800 --> 00:04:45.840 third paradigm really began, I guess, sort of,  in the ’50s and ’60s, the development of digital   00:04:45.840 --> 00:04:50.000 computers. And, actually, the very first use  of digital computers was to simulate physics,   00:04:50.000 --> 00:04:55.800 and it's been at the core of digital computing  right up to the present day. And so what you're   00:04:55.800 --> 00:05:01.360 doing there is using a computer to go with a  numerical algorithm to solve those very simple   00:05:01.360 --> 00:05:06.520 equations but solve them in a practical setting.  And so that's, I’ll refer to that as simulation.   00:05:06.520 --> 00:05:12.200 That's the third paradigm. And that's proven  to be tremendously powerful. If you look up the   00:05:12.200 --> 00:05:16.920 weather forecast on your phone today, it's done  by numerical weather forecasting, solving in those   00:05:16.920 --> 00:05:22.280 case Navier-Stokes equations using big numerical  simulators. What Jim Gray observed, though,   00:05:22.280 --> 00:05:28.160 really emerging at the beginning of the 21st  century was what he called the fourth paradigm,   00:05:28.160 --> 00:05:35.040 or data-intensive scientific discovery. So this  is the era of big data. Think of particle physics   00:05:35.040 --> 00:05:42.320 at the CERN accelerator, for example, generating  colossal amounts of data in real time. And that   00:05:42.320 --> 00:05:46.680 data can then be processed and filtered. We can do  statistics on it. But of course, we can do machine   00:05:46.680 --> 00:05:53.280 learning on that data. And so machine learning  feeds off large data. And so the fourth paradigm   00:05:53.280 --> 00:05:59.240 really is dominated today by machine learning.  And again that remains tremendously important.  00:05:59.240 --> 00:06:03.280 What I noticed, though, is that there's  again another framework. We call it the fifth   00:06:03.280 --> 00:06:08.440 paradigm. Again, it goes back to those fundamental  equations. But again, it's driven by computation,   00:06:08.440 --> 00:06:16.360 and it's the idea that we can train machine  learning systems not using the empirical data   00:06:16.360 --> 00:06:21.240 of the fourth paradigm but instead using the  results of simulation. So the output of the   00:06:21.240 --> 00:06:27.160 third paradigm. So think of it this way. You want  to predict the property of some molecule, let's   00:06:27.160 --> 00:06:32.080 say. You could in principle solve Schrödinger’s  equation on a digital computer; it’d be very   00:06:32.080 --> 00:06:36.200 expensive. And let's say you want to screen  hundreds of millions of molecules. That's going   00:06:36.200 --> 00:06:42.800 to get far too costly. So instead, what you can  do is have a mindset shift. You can think of that   00:06:42.800 --> 00:06:48.600 simulator not as a tool to predict the molecule’s  properties directly but instead as a way of   00:06:48.600 --> 00:06:54.200 generating synthetic training data. And then you  use that training data to train a deep learning   00:06:54.200 --> 00:07:00.800 system to give what I like to call an emulator,  an emulator of the simulator. Once it's trained,   00:07:00.800 --> 00:07:07.760 that emulator is fast. It's usually three to four  orders of magnitude faster than the simulator. So   00:07:07.760 --> 00:07:12.320 if you're going to do something over and over  again, that three-to-four-order-of-magnitude   00:07:12.320 --> 00:07:17.160 acceleration is tremendously disruptive. And  what's really interesting is we see that fifth   00:07:17.160 --> 00:07:21.800 paradigm occur in many, many different places.  The idea goes back a long way. The, actually,   00:07:21.800 --> 00:07:27.640 the last project that I worked on before I left  the fusion program was to do what was the world's   00:07:27.640 --> 00:07:35.200 first-ever real-time control of a tokamak fusion  plasma using a neural net and the computers of the   00:07:35.200 --> 00:07:40.680 day. But the processors were just far too slow,  long before GPUs, and so on. And so it wasn't   00:07:40.680 --> 00:07:45.240 possible to solve the equations. In that case,  it was called the Grad-Shafranov equation. Again,   00:07:45.240 --> 00:07:49.360 a simple differential equation you could write  on a T-shirt, but solving it was expensive on   00:07:49.360 --> 00:07:56.280 a computer. We were about a million times too slow  to solve it directly in real time. And so instead,   00:07:56.280 --> 00:08:01.200 we generated lots and lots of solutions. We  used those solutions to train a very simple   00:08:01.200 --> 00:08:05.720 neural network, not a deep network, just a  simple two-layer network back in the day,   00:08:05.720 --> 00:08:09.600 and then we implemented that in special hardware  and did real-time feedback control. So that was an   00:08:09.600 --> 00:08:14.360 example of the fifth paradigm from, you know,  a quarter of a century ago. But of course,   00:08:14.360 --> 00:08:21.000 deep learning just tremendously expands  the range of applicability. So today we're   00:08:21.000 --> 00:08:27.000 using the fifth paradigm in many, many different  scenarios. And time and time again, we see these   00:08:27.000 --> 00:08:32.560 four-orders-of-magnitude acceleration. So I think  it's worthy of thinking of that as a new paradigm   00:08:32.560 --> 00:08:37.240 because it's so pervasive and so ubiquitous. STRICKLAND: So how do you identify fields of   00:08:37.240 --> 00:08:41.440 science and particular problems that are  amenable to this kind of AI assistance?   00:08:41.440 --> 00:08:44.980 Is it all about availability of data  or the need for that kind of speed up?  00:08:44.980 --> 00:08:49.000 BISHOP: So there are lots of factors that  go into this. And when I think about AI for   00:08:49.000 --> 00:08:54.240 Science actually, the space of opportunity  is colossal because science is, science is   00:08:54.240 --> 00:08:59.800 really just understanding more about the world  around us. And so the range of possibilities is   00:08:59.800 --> 00:09:05.840 daunting really. So in choosing what to work on,  I think there are several factors. Yes, of course,   00:09:05.840 --> 00:09:10.560 data is important, but very interestingly, we  can use experimental data or we can generate   00:09:10.560 --> 00:09:14.960 synthetic data by running simulators. So we're a  big fan of the fifth paradigm. But I think another   00:09:14.960 --> 00:09:20.480 factor—and this is particularly at Microsoft—is  thinking about, how can we have real-world impact   00:09:20.480 --> 00:09:24.680 at scale? Because that's our job, is to make the  world a better place and to do so at a planetary   00:09:24.680 --> 00:09:31.960 scale. And so we've settled on, for the most part,  working at the molecular level. So if you think   00:09:31.960 --> 00:09:36.720 about the number of different ways of combining  atoms together to make new stable configurations   00:09:36.720 --> 00:09:43.800 of atoms, it’s gargantuan. I mean, the number of  just small molecules, small organic molecules,   00:09:43.800 --> 00:09:48.200 that are potential drug candidates is about 10  to the power of 60. It's about the same as the   00:09:48.200 --> 00:09:53.200 number of atoms in the solar system. The number  of proteins, maybe the fourth power of the number   00:09:53.200 --> 00:09:58.080 of atoms in the universe, or something crazy. So  you've got this gargantuan space to search, and   00:09:58.080 --> 00:10:03.440 within that space, for sure, there'll be all sorts  of interesting molecules, materials, new drugs,   00:10:03.440 --> 00:10:08.280 new therapies, new materials for carbon capture,  new kinds of batteries, new photovoltaics. The   00:10:08.280 --> 00:10:12.840 list is endless because everything around us  is made of atoms, including our own bodies.   00:10:12.840 --> 00:10:18.240 So the potential just in the molecular space is  gargantuan. And so that's why we focus there.  00:10:18.240 --> 00:10:21.960 STRICKLAND: It's a big focus. [LAUGHTER] BISHOP: It's a broad focus, still, yes.  00:10:21.960 --> 00:10:27.600 STRICKLAND: So let's take one of these case  studies then. In a project on drug discovery,   00:10:27.600 --> 00:10:33.440 you worked with the Global Health Drug Discovery  Institute on molecules that would interact with   00:10:33.440 --> 00:10:40.080 tuberculosis and coronaviruses, I think. And you  found, I think, candidate molecules in five months   00:10:40.080 --> 00:10:46.040 instead of several years. Can you talk about  what models you used in this work and how they   00:10:46.040 --> 00:10:50.520 helped you get this vastly sped up process? BISHOP: Sure. Yes. We're very proud of this   00:10:50.520 --> 00:10:55.640 project. We're working with the Gates Foundation  and the Global Health Drug Discovery Institute to   00:10:55.640 --> 00:10:59.240 look at particularly diseases that affect  low-income countries like tuberculosis.   00:11:00.200 --> 00:11:04.400 And in terms of the models we use, I think we're  all familiar with a large language model. We train   00:11:04.400 --> 00:11:09.000 it on a sequence of words or sequence of word  tokens, and it's trained to predict the next   00:11:09.000 --> 00:11:13.880 token. We can do a similar thing, but instead of  learning the language of humans, we can learn the   00:11:13.880 --> 00:11:20.120 language of nature. So in particular, what we're  looking for here is a small organic molecule   00:11:20.120 --> 00:11:24.840 that we could synthesize in a laboratory that will  bind with a particular target protein. It's called   00:11:24.840 --> 00:11:30.800 ClpP. And by interfering with that protein, we can  arrest the process of tuberculosis. So the goal is   00:11:30.800 --> 00:11:36.320 to search that space of 10 to the 60 molecules and  find a new one that has the right properties. Now,   00:11:36.320 --> 00:11:41.200 the way we do this is to train something that's  essentially a transformer. So it looks like a   00:11:41.200 --> 00:11:46.520 language model, but the language it's trained  on is a thing called SMILES strings. It's an   00:11:46.520 --> 00:11:49.960 idea that's been around in chemistry for  a long time. It's just a way of taking a   00:11:49.960 --> 00:11:55.280 three-dimensional molecule and representing it as  a one-dimensional sequence of characters. So this   00:11:55.280 --> 00:12:01.680 is perfect for feeding into a language model. So  we take a transformer and we train it on a large   00:12:01.680 --> 00:12:06.680 database of small organic molecules that are, sort  of, typical of the kinds of things you might see   00:12:06.680 --> 00:12:12.840 in the space of drug molecules. Once that's been  trained, we can now run it generatively. And it   00:12:12.840 --> 00:12:17.240 will output new molecules. Now, we don't just  want to generate molecules at random because   00:12:17.240 --> 00:12:22.480 that doesn't help. We want to generate molecules  that bind to this particular binding site on this   00:12:22.480 --> 00:12:27.440 particular protein. So the next step is we  have to tell the model about the protein and   00:12:27.440 --> 00:12:33.320 the protein binding site. And we do that by giving  it information about not actually—well, we do tell   00:12:33.320 --> 00:12:37.200 it about the whole protein, but we especially  give it information about the three-dimensional   00:12:37.200 --> 00:12:42.600 geometry of the binding site. So we tell about  the locations of the atoms that are in the binding   00:12:42.600 --> 00:12:47.760 site. And we do this in a way that satisfies  certain physics constraints, sort of, equivariance   00:12:47.760 --> 00:12:52.240 properties, it's called. So if you think about a  molecule, if I rotate the molecule in space, the   00:12:52.240 --> 00:12:56.280 positions of all the atoms change in a complicated  way. But it's the same molecule; it has the same   00:12:56.280 --> 00:13:01.480 energy and other properties and so on. So we need  the right kind of representation. That's then fed   00:13:01.480 --> 00:13:06.440 into this transformer using a technique called  cross-attention. So internally, the transformer   00:13:06.440 --> 00:13:11.120 uses self-attention to look at the history  of tokens, but it can now use cross-attention   00:13:11.120 --> 00:13:15.560 to look at another model that understands the  proteins. But even that's not enough. Because   00:13:15.560 --> 00:13:21.120 in discovering drugs and exploring this gargantuan  space and looking for these needles in a haystack,   00:13:21.120 --> 00:13:25.680 what typically happens [is] you find a hit,  a molecule that binds, but now you want to   00:13:25.680 --> 00:13:30.120 optimize it. You want to make lots of small  variations of that molecule in order to make   00:13:30.120 --> 00:13:34.800 it better and better at binding. So the third  piece of the architecture is another module, a   00:13:34.800 --> 00:13:40.440 thing called a variational autoencoder, that again  uses deep learning. But this time, it can take as   00:13:40.440 --> 00:13:46.960 input an organic molecule that is already known,  a hit that's already known to bind to the site,   00:13:46.960 --> 00:13:53.160 and that again is fed in through cross-attention.  And now the SMILES autoregressive model can now   00:13:53.160 --> 00:13:57.560 generate a molecule that's an improvement  on the starting molecule and knows about the   00:13:57.560 --> 00:14:02.880 protein binding. And so what we do is, we start  off with the state-of-the-art molecule. And the   00:14:02.880 --> 00:14:08.120 best example we found is one that's more than  two orders of magnitude stronger binding affinity   00:14:08.120 --> 00:14:12.520 to the binding pocket, which is a tremendous  advance; it’s the state of the art in addressing   00:14:12.520 --> 00:14:17.080 tuberculosis. And of course, the exciting thing  is that this is tested in the laboratory. So this   00:14:17.080 --> 00:14:21.320 is not just a computer experiment in some sort  of benchmark or whatever. We sent a description   00:14:21.320 --> 00:14:26.480 of the molecule to the laboratories at GHDDI.  They synthesized a molecule, characterized it,   00:14:26.480 --> 00:14:30.800 measured its binding property, and said, well,  hey, this is a new state of the art for this   00:14:30.800 --> 00:14:35.920 target protein. So we're continuing to work with  them to further refine this. There are obviously   00:14:35.920 --> 00:14:40.040 quite a few more steps. If you know about the  drug discovery process, there’s a lot of hurdles   00:14:40.040 --> 00:14:43.520 you have to get through, including, of course,  very important clinical trials, before you have   00:14:43.520 --> 00:14:49.200 something that can actually be used in humans.  But we're already hugely excited about the fact   00:14:49.200 --> 00:14:53.720 that we were able to make such a big advance  so quickly, in such a short amount of time,   00:14:53.720 --> 00:14:59.040 compared to the usual drug discovery process. STRICKLAND: And while you were looking for that   00:14:59.040 --> 00:15:04.720 molecule that had the proper characteristics,  were you also determining whether it could be   00:15:04.720 --> 00:15:08.240 manufactured easily, like trying to think about  practical realities of bringing this thing out   00:15:08.240 --> 00:15:11.320 of the computer and into the lab? BISHOP: Great question. I mean,   00:15:11.320 --> 00:15:16.240 you're hinting there at the fact the discovery  process, of course, is a long pipeline. You start   00:15:16.240 --> 00:15:20.480 with the protein. You have to find a molecule that  binds. You then refine the molecule. Now you have   00:15:20.480 --> 00:15:25.440 to look at ADMET, you know, the absorption,  metabolism, and excretion and so on of the   00:15:25.440 --> 00:15:30.000 molecule. Also make sure that it's not toxic.  But then you need to be able to synthesize it.   00:15:30.000 --> 00:15:33.400 It's no good if nobody can make this molecule.  So you have to look at that. So, actually,   00:15:33.400 --> 00:15:38.720 in the AI for Science team, we look at all of  these aspects of that drug discovery process.   00:15:38.720 --> 00:15:42.320 And we find particular areas, especially where  there’s, sort of, low-hanging fruit where we can   00:15:42.320 --> 00:15:47.640 see that deep learning can make a big impact. It  doesn't necessarily help much to take a very easy,   00:15:47.640 --> 00:15:51.280 fast piece of the pipeline and go work on that.  You want to understand, what are the bottlenecks,   00:15:51.280 --> 00:15:54.560 and can we really unlock those with deep  learning? So we're very interested in that   00:15:54.560 --> 00:15:59.120 whole process. It’s a fascinating problem.  You've got a gargantuan search space,   00:15:59.120 --> 00:16:03.280 and yet you have so many different constraints  that need to be met. And deep learning just feels   00:16:03.280 --> 00:16:07.520 like the perfect tool to go after this problem. STRICKLAND: When you talk to the scientists   00:16:07.520 --> 00:16:11.200 that you collaborate with, is AI  changing the kinds of questions   00:16:11.200 --> 00:16:16.520 that they are able to ask? That they want to ask? BISHOP: Oh, for sure. And it's really empowering.   00:16:16.520 --> 00:16:22.400 It's enabling those working in the drug discovery  space to, I think, to think in a much more   00:16:22.400 --> 00:16:27.200 expansive way. If you think about just the kind  of acceleration that I talked about from the fifth   00:16:27.200 --> 00:16:33.080 paradigm, if you go to four-order-of-magnitude  acceleration, OK, it may not sound like much of   00:16:33.080 --> 00:16:38.600 a dent onto the 10 to the power 60 space, but now  when you're exploring variants of molecules and so   00:16:38.600 --> 00:16:43.520 on, the ability to explore that space orders  of magnitude faster allows you to think much   00:16:43.520 --> 00:16:49.120 more creatively, allows you to think in a more  expansive way about how much of that space you can   00:16:49.120 --> 00:16:53.280 explore and how efficiently you can explore it.  So I think it really is opening up new horizons,   00:16:53.280 --> 00:16:58.160 and certainly, we have an exciting partnership  with Novartis. We've been working with them for   00:16:58.160 --> 00:17:04.280 the last five years, and they've been deploying  some of our techniques and models in practice   00:17:04.280 --> 00:17:09.880 for their drug discovery pipeline. We get a lot  of great feedback from them about how exciting   00:17:09.880 --> 00:17:13.640 they're finding these techniques to use in  practice because it is changing the way they   00:17:13.640 --> 00:17:18.840 go about doing the drug discovery process. STRICKLAND: To jump to one other case study,   00:17:18.840 --> 00:17:22.960 we don't have to go into great detail on it,  but I'm very curious about your Project Aurora,   00:17:22.960 --> 00:17:27.480 this foundation model for state-of-the-art  weather forecasting that, I believe, is 5,000   00:17:27.480 --> 00:17:32.760 times faster than traditional physics-based  methods. Can you talk a little bit about how   00:17:32.760 --> 00:17:38.200 that project is evolving, how you imagine these  AI forecasting models working with traditional   00:17:38.200 --> 00:17:42.640 forecasting models, perhaps, or replacing them? BISHOP: Yes. So I said most of what we do is   00:17:42.640 --> 00:17:47.320 down at the molecular level. So this is one of the  exceptions. So this is really at the global level,   00:17:47.320 --> 00:17:52.200 the planetary level. Again, it's a beautiful  example of the fifth paradigm because the way   00:17:52.200 --> 00:17:56.240 forecasting has been done for a number of  decades now and the way most forecasting is   00:17:56.240 --> 00:17:59.920 done at the moment is through what's called  numerical weather prediction. So again,   00:17:59.920 --> 00:18:05.200 you have these simple equations. It's no longer  Schrödinger’s equation of atomic physics. It's   00:18:05.200 --> 00:18:10.320 now Navier–Stokes equations of fluid flows and  a whole bunch of other equations that describe   00:18:10.320 --> 00:18:15.160 moisture in the atmosphere and the weather  and so on. And those equations are solved   00:18:15.160 --> 00:18:21.800 on a supercomputer. And again, we can think  of that numerical simulator now not just as   00:18:21.800 --> 00:18:26.320 the way you're going to do the forecasting but  actually as the way to generate training data   00:18:26.320 --> 00:18:30.520 for a deep learning emulator. So several  groups have been exploring this over the   00:18:30.520 --> 00:18:35.640 last couple of years. And again, we see this  very robust three-to-four-order-of-magnitude   00:18:35.640 --> 00:18:41.240 acceleration. But what's really interesting about  Aurora, it's the world's first foundation model,   00:18:41.240 --> 00:18:47.000 so instead of just building an emulator of  a particular numerical weather simulator,   00:18:47.000 --> 00:18:53.720 which is already very interesting, we trained  Aurora on a much more diverse set of data and   00:18:53.720 --> 00:18:58.720 really trying to force it not just to emulate  a particular simulator but really, as it were,   00:18:59.360 --> 00:19:05.440 understand or model the fundamental equations of  fluid flows in the Earth's atmosphere. And then   00:19:05.440 --> 00:19:10.280 the reason we want to do this is because we now  want to take that foundation model and fine-tune   00:19:10.280 --> 00:19:16.040 it to other downstream applications where there’s  much less data. So one example would be pollution   00:19:16.040 --> 00:19:21.960 flow. So obviously the flow of pollution around  the atmosphere is extremely important. But the   00:19:21.960 --> 00:19:26.200 data is far more sparse. There are far fewer  sensors for pollution than there are for, sort of,   00:19:26.200 --> 00:19:32.760 wind and rain and temperature and so on. And so we  were able to achieve state-of-the-art performance   00:19:32.760 --> 00:19:39.000 in modeling the flow of pollution by leveraging  huge data and building this foundation model and   00:19:39.000 --> 00:19:44.520 then using relatively little data, our pollution  monitoring, to build that downstream fine-tuned   00:19:44.520 --> 00:19:49.680 model. So beautiful example of a foundation model. STRICKLAND: That is a cool example. And finally,   00:19:49.680 --> 00:19:53.320 just to wrap up, what have you seen or heard  at NeurIPS that’s gotten you excited? What   00:19:53.320 --> 00:19:56.320 kind of trends are in the air? What’s the buzz? BISHOP: Oh, that’s a great question. I mean,   00:19:56.320 --> 00:20:00.280 it's such a huge conference. There's something  like 17,000 people or so here this year,   00:20:00.280 --> 00:20:05.040 I've heard. I think, you know, one of the things  that's happened so far that's actually given me an   00:20:05.040 --> 00:20:10.320 enormous amount of energy wasn’t just a technical  talk. It was actually an event we had on the first   00:20:10.320 --> 00:20:15.720 day called Women in Machine Learning. And I  was a mentor on one of the mentorship tables,   00:20:15.720 --> 00:20:21.320 and I found it very energizing just to meet  so many people, early-career-stage people,   00:20:21.320 --> 00:20:26.440 who were very excited about AI for Science and  realizing that, you know, it's not just that I   00:20:26.440 --> 00:20:30.560 think AI for Science is important. A lot of  people are moving into this field now. It is   00:20:30.560 --> 00:20:35.520 a big frontier for AI. I'm a little biased,  perhaps. I think that it's the most important   00:20:35.520 --> 00:20:39.840 application area. Intellectually, it's very  exciting because we get to deal with science   00:20:39.840 --> 00:20:45.880 as well as machine learning. But also if you think  about [it], science is really about learning more   00:20:45.880 --> 00:20:50.680 about the world. And once we learn more about  the world, we can then develop aquaculture;   00:20:50.680 --> 00:20:54.120 we can develop the steam engine; we can develop  silicon chips; we can change the world. We can   00:20:54.120 --> 00:20:58.720 save lives and make the world a better place. And  so I think it's the most fundamental undertaking   00:20:58.720 --> 00:21:03.720 we have in AI for Science and the thing I  loved about the Women in Machine Learning   00:21:03.720 --> 00:21:08.000 event is that the AI for Science table was  just completely swamped with all of these   00:21:08.000 --> 00:21:12.160 people at early stages of their career, either  already working in this field and doing PhDs   00:21:12.160 --> 00:21:16.280 or wanting to get into it. That was very exciting. STRICKLAND: That is really exciting and inspiring,   00:21:16.280 --> 00:21:20.520 and it gives me a lot of hope. Well, Chris  Bishop, thank you so much for joining us   00:21:20.520 --> 00:21:24.061 today and thanks for a great conversation. BISHOP: Thank you. I really appreciate it.  00:21:24.061 --> 00:21:25.080 [MUSIC] STRICKLAND: And to our listeners,   00:21:25.080 --> 00:21:28.600 thanks for tuning in. If you want to  learn more about research at Microsoft,   00:21:28.600 --> 00:21:33.560 you can check out the Microsoft Research  website at microsoft.com/research. Until   00:21:33.560 --> 00:21:50.004 next time. [MUSIC FADES]