00:00:00.983 --> 00:00:01.711 [MUSIC]  00:00:01.711 --> 00:00:08.720 AMBER TINGLE: Welcome to Abstracts, a Microsoft  Research Podcast that puts the spotlight on   00:00:08.720 --> 00:00:16.240 world-class research in brief. I’m Amber Tingle.  In this series, members of the research community   00:00:16.240 --> 00:00:24.240 at Microsoft give us a quick snapshot—or a podcast  abstract—of their new and noteworthy papers. 00:00:24.240 --> 00:00:25.120 [MUSIC FADES] 00:00:25.120 --> 00:00:31.920 Our guests today are Megan Stanley and Wessel  Bruinsma. They are both senior researchers   00:00:31.920 --> 00:00:38.080 within the Microsoft Research AI for Science  initiative. They are also two of the coauthors   00:00:38.080 --> 00:00:44.640 on a new Nature publication called “A  Foundation Model for the Earth System.” 00:00:44.640 --> 00:00:48.000 This is such exciting work  about environmental forecasting,   00:00:48.000 --> 00:00:51.040 so we're happy to have the  two of you join us today. 00:00:51.040 --> 00:00:52.960 Megan and Wessel, welcome. 00:00:52.960 --> 00:00:56.207 MEGAN STANLEY: Thank you.  Thanks. Great to be here. 00:00:56.207 --> 00:00:58.480 WESSEL BRUINSMA: Thanks. TINGLE: Let's jump right in. Wessel, 00:00:58.480 --> 00:01:03.680 share a bit about the problem your research  addresses and why this work is so important. 00:01:03.680 --> 00:01:08.240 BRUINSMA: I think we're all very much aware of the  revolution that's happening in the space of large   00:01:08.240 --> 00:01:13.680 language models, which have just become so strong.  What's perhaps lesser well-known is that machine   00:01:13.680 --> 00:01:18.800 learning models have also started to revolutionize  this field of weather prediction. Whereas   00:01:18.800 --> 00:01:23.520 traditional weather prediction models, based on  physical laws, used to be the state of the art,   00:01:23.520 --> 00:01:28.320 these traditional models are now challenged  and often even outperformed by AI models. 00:01:28.320 --> 00:01:32.960 This advancement is super impressive and  really a big deal. Mostly because AI weather   00:01:32.960 --> 00:01:36.400 forecasting models are computationally  much more efficient and can even be   00:01:36.400 --> 00:01:40.400 more accurate. What's unfortunate  though, about this big step forward,   00:01:40.400 --> 00:01:45.040 is that these developments are mostly limited  to the setting of weather forecasting. 00:01:45.040 --> 00:01:48.880 Weather forecasting is very important,  obviously, but there are many other   00:01:48.880 --> 00:01:54.160 important environmental forecasting problems  out there, such as air pollution forecasting   00:01:54.160 --> 00:02:00.400 or ocean wave forecasting. We have developed a  model, named Aurora, which really kicks the AI   00:02:00.400 --> 00:02:05.280 revolution in weather forecasting into the  next gear by extending these advancements   00:02:05.280 --> 00:02:10.560 to other environmental forecasting fields,  too. With Aurora, we're now able to produce   00:02:10.560 --> 00:02:15.960 state-of-the-art air pollution forecasts using  an AI approach. And that wasn't possible before! 00:02:15.960 --> 00:02:19.440 TINGLE: Megan, how does this approach differ from   00:02:19.440 --> 00:02:24.280 or build on work that's already been  done in the atmospheric sciences? 00:02:24.280 --> 00:02:28.480 STANLEY: Current approaches have really  focused training very specifically on   00:02:28.480 --> 00:02:34.080 weather forecasting models. And in contrast,  with Aurora, what we've attempted to do is   00:02:34.080 --> 00:02:38.400 train a so-called foundation model for  the Earth system. In the first step,   00:02:38.400 --> 00:02:44.240 we train Aurora on a vast body of Earth  system data. This is our pretraining step. 00:02:44.240 --> 00:02:49.520 And when I say a vast body of data, I really do  mean a lot. And the purpose of this pretraining   00:02:49.520 --> 00:02:54.640 is to let Aurora, kind of, learn some  general-purpose representation of the   00:02:54.640 --> 00:02:59.200 dynamics that govern the Earth system.  But then once we've pretrained Aurora,   00:02:59.200 --> 00:03:04.000 and this really is the crux of this, the reason  why we're doing this project, is after the model   00:03:04.000 --> 00:03:09.280 has been pretrained, it can leverage this  learned general-purpose representation and   00:03:09.280 --> 00:03:15.920 efficiently adapt to new tasks, new domains,  new variables. And this is called fine-tuning. 00:03:15.920 --> 00:03:19.920 The idea is that the model really uses  the learned representation to perform   00:03:19.920 --> 00:03:25.680 this adaptation very efficiently, which  basically means Aurora is a powerful,   00:03:25.680 --> 00:03:31.200 flexible model that can relatively cheaply be  adapted to any environmental forecasting task. 00:03:31.200 --> 00:03:34.320 TINGLE: Wessel, can you tell us about your   00:03:34.320 --> 00:03:38.200 methodology? How did you  all conduct this research? 00:03:38.200 --> 00:03:44.240 BRUINSMA: While approaches so far have trained  models on primarily one particular data set,   00:03:44.240 --> 00:03:49.440 this one dataset is very large, which makes  it possible to train very good models. But it   00:03:49.440 --> 00:03:54.560 does remain only one dataset, and that's not  very diverse. In the domain of environmental   00:03:54.560 --> 00:03:59.760 forecasting, we have really tried to push the  limits of scaling to large data by training   00:03:59.760 --> 00:04:05.760 Aurora on not just this one large dataset, but  on as many very large datasets as we could find. 00:04:05.760 --> 00:04:10.080 These datasets are a combination of estimates  of the historical state of the world,   00:04:10.080 --> 00:04:15.040 forecasts by other models, climate simulations,  and more. We've been able to show that training   00:04:15.040 --> 00:04:19.760 on not just more data but more diverse  data helps the model achieve even better   00:04:19.760 --> 00:04:24.080 performance. Showing this is difficult  because there is just so much data. 00:04:24.080 --> 00:04:28.320 In addition to scaling to more and more  diverse data, we also increased the size   00:04:28.320 --> 00:04:34.080 of the model as much as we could. Here we found  that bigger models, despite being slower to run,   00:04:34.080 --> 00:04:39.200 make more efficient use of computational  resources. It's cheaper to train a good big   00:04:39.200 --> 00:04:44.400 model than a good small model. The mantra of  this project was to really keep it simple and   00:04:44.400 --> 00:04:50.240 to scale to simultaneously very large and, more  importantly, diverse data and large model size. 00:04:50.240 --> 00:04:53.280 TINGLE: So, Megan, what were your major   00:04:53.280 --> 00:04:56.871 findings? And we know they're major  because they're in Nature. [LAUGHS] 00:04:56.871 --> 00:05:02.320 STANLEY: Yeah, [LAUGHS] I guess they really are.  So the main outcome of this project is we were   00:05:02.320 --> 00:05:08.480 actually able to train a single foundation model  that achieves state-of-the-art performance in   00:05:08.480 --> 00:05:12.880 four different domains. Air pollution  forecasting. For example, predicting   00:05:12.880 --> 00:05:18.400 particulate matter near the surface or ozone  in the atmosphere. Ocean wave forecasting,   00:05:18.400 --> 00:05:21.680 which is critical for planning shipping routes. 00:05:21.680 --> 00:05:23.600 Tropical cyclone track forecasting,   00:05:23.600 --> 00:05:29.200 so that means being able to predict where  a hurricane or a typhoon is expected to go,   00:05:29.200 --> 00:05:34.320 which is obviously incredibly important, and  very high-resolution weather forecasting. 00:05:34.320 --> 00:05:38.320 And I've, kind of, named these forecasting  domains as if they're just items in a list,   00:05:38.320 --> 00:05:43.040 but in every single one, Aurora really  pushed the limits of what is possible   00:05:43.040 --> 00:05:46.240 with AI models. And we're really proud of that. 00:05:46.240 --> 00:05:51.680 But perhaps, kind of, you know, to my mind, the  key takeaway here is that the foundation model   00:05:51.680 --> 00:05:56.960 approach actually works. So what we have shown  is it's possible to actually train some kind   00:05:56.960 --> 00:06:02.720 of general model, a foundation model, and then  adapt it to a wide variety of environmental tasks.   00:06:02.720 --> 00:06:07.440 Now we definitely do not claim that Aurora  is some kind of ultimate environmental   00:06:07.440 --> 00:06:12.240 forecasting model. We are sure that the  model and the pretraining procedure can   00:06:12.240 --> 00:06:18.560 actually be improved. But, nevertheless,  we've shown that this approach works for   00:06:18.560 --> 00:06:22.920 environmental forecasting. It really holds  massive promise, and that's incredibly cool. 00:06:22.920 --> 00:06:29.840 TINGLE: Wessel, what do you think will  be the real-world impact of this work? 00:06:29.840 --> 00:06:33.600 BRUINSMA: Well, for applications that  we mentioned, which are air pollution   00:06:33.600 --> 00:06:37.760 forecasting, ocean wave forecasting,  tropical cyclone track forecasting,   00:06:37.760 --> 00:06:43.200 and very high-resolution weather forecasting,  Aurora could today be deployed in real-time   00:06:43.200 --> 00:06:47.440 systems to produce near real-time  forecasts. And, you know, in fact,   00:06:47.440 --> 00:06:53.040 it already is. You can view real-time weather  forecasts by the high-resolution version of   00:06:53.040 --> 00:06:55.920 the model on the website of ECMWF (European  Centre for Medium-Range Weather Forecasts). 00:06:55.920 --> 00:07:01.040 But what's remarkable is that every of these  applications took a small team of engineers   00:07:01.040 --> 00:07:06.000 about four to eight weeks to fully execute. You  should compare this to a typical development   00:07:06.000 --> 00:07:11.520 timeline for more traditional models, which  can be on the order of multiple years. Using   00:07:11.520 --> 00:07:16.400 the pretraining fine-tuning approach that we  used for Aurora, we might see significantly   00:07:16.400 --> 00:07:21.760 accelerated development cycles for environmental  forecasting problems. And that's exciting. 00:07:21.760 --> 00:07:28.720 TINGLE: Megan, if our listeners only walk away  from this conversation with one key talking point,   00:07:28.720 --> 00:07:33.240 what would you like that to be? What  should we remember about this paper? 00:07:33.240 --> 00:07:37.920 STANLEY: The biggest takeaway is that  the pretraining fine-tuning paradigm,   00:07:37.920 --> 00:07:42.880 it really works for environmental forecasting,  right? So you can train a foundational model,   00:07:42.880 --> 00:07:47.040 it learns some kind of general-purpose  representation of the Earth system dynamics,   00:07:47.040 --> 00:07:53.280 and this representation boosts performance in a  wide variety of forecasting tasks. But we really   00:07:53.280 --> 00:07:58.320 want to emphasize that Aurora only scratches  the surface of what's actually possible. 00:07:58.320 --> 00:08:03.440 So there are many more applications to explore  than the four we've mentioned. And undoubtedly,   00:08:03.440 --> 00:08:05.760 the model and pretraining procedure can actually   00:08:05.760 --> 00:08:10.880 be improved. So we're really excited to  see what the next few years will bring. 00:08:10.880 --> 00:08:14.880 TINGLE: Wessel, tell us more about  those opportunities and unanswered   00:08:14.880 --> 00:08:19.800 questions. What's next on the research  agenda in environmental prediction? 00:08:19.800 --> 00:08:25.280 BRUINSMA: Well, Aurora has two main  limitations. The first is that the   00:08:25.280 --> 00:08:31.120 model produces only deterministic predictions,  by which I mean a single predicted value. For   00:08:31.120 --> 00:08:36.400 variables like temperature, this is mostly  fine. But other variables like precipitation,   00:08:36.400 --> 00:08:40.880 they are inherently some kind of stochastic.  For these variables, we really want to assign   00:08:40.880 --> 00:08:46.560 probabilities to different levels of precipitation  rather than predicting only a single value. 00:08:46.560 --> 00:08:51.760 An extension of Aurora to allow this sort  of prediction would be a great next step. 00:08:51.760 --> 00:08:57.280 The second limitation is that Aurora depends on  a procedure called assimilation. Assimilation   00:08:57.280 --> 00:09:02.480 attempts to create a starting point for the model  from real-world observations, such as from weather   00:09:02.480 --> 00:09:08.160 stations and satellites. The model then takes the  starting point and uses it to make predictions.   00:09:08.720 --> 00:09:11.760 Unfortunately, assimilation is super expensive,   00:09:11.760 --> 00:09:15.920 so it would be great if we could  somehow circumvent the need for it. 00:09:15.920 --> 00:09:20.160 Finally, what we find really important is  to make our advancements available to the   00:09:20.160 --> 00:09:21.432 community. [MUSIC] 00:09:21.432 --> 00:09:23.520 TINGLE: Great. Megan and Wessel,   00:09:23.520 --> 00:09:26.240 thanks for joining us today on  the Microsoft Research Podcast. 00:09:26.240 --> 00:09:27.520 BRUINSMA: Thanks for having us. 00:09:27.520 --> 00:09:29.120 STANLEY: Yeah, thank you. It's been great. 00:09:29.120 --> 00:09:35.200 TINGLE: You can check out the Aurora model on  Azure AI Foundry. You can read the entire paper,   00:09:35.200 --> 00:09:40.880 “A Foundation Model for the Earth  System,” at aka.ms/abstracts. And   00:09:40.880 --> 00:09:44.480 you'll certainly find it  on the Nature website, too. 00:09:44.480 --> 00:09:52.320 Thank you so much for tuning in to  Abstracts today. Until next time. 00:09:52.320 --> 00:09:53.169 [MUSIC FADES]