00:00:00.034 --> 00:00:03.233 [MUSIC PLAYS] 00:00:03.233 --> 00:00:07.160 GRETCHEN HUIZINGA: Welcome to Abstracts,  a Microsoft Research Podcast that puts the   00:00:07.160 --> 00:00:11.931 spotlight on world-class research in   brief. I’m Dr. Gretchen Huizinga. 00:00:15.211 --> 00:00:20.080 In this series, members of the research community at Microsoft give us a quick snapshot—or   00:00:20.080 --> 00:00:23.829 a podcast abstract—of their  new and noteworthy papers. 00:00:23.829 --> 00:00:25.200 [MUSIC FADES] 00:00:25.200 --> 00:00:29.680 I'm here today with Dr. Arindam Mitra, a  senior researcher at Microsoft Research   00:00:29.680 --> 00:00:35.440 and the lead researcher for Microsoft's Orca  project. Dr. Mitra is coauthor of a paper   00:00:35.440 --> 00:00:41.280 called “AgentInstruct: Toward Generative  Teaching with Agentic Flows.” Arindam,   00:00:41.280 --> 00:00:43.665 it's a pleasure to have you on Abstracts today. 00:00:43.665 --> 00:00:44.720 ARINDAM MITRA: Thank you, Gretchen. 00:00:44.720 --> 00:00:48.440 HUIZINGA: So let's start with  a brief overview of your paper.   00:00:48.440 --> 00:00:51.900 What problem does your research  address, and why does it matter? 00:00:51.900 --> 00:00:56.920 MITRA: So the post-training phase is very  important for language models. You can really   00:00:56.920 --> 00:01:03.640 improve the model a lot by creating high-quality  synthetic data. The problem is, however, though,   00:01:03.640 --> 00:01:09.160 high-quality synthetic data creation requires  lots of human effort and expertise. The problem   00:01:09.160 --> 00:01:15.520 that we're trying to tackle is, how do you reduce  human effort? How can you create high-quality data   00:01:15.520 --> 00:01:21.200 with really low amount of human effort? When you  have a language model and, let's say, you want   00:01:21.200 --> 00:01:26.520 to apply it somewhere, you might have to train a  generic model before. Which could be small or big.   00:01:26.520 --> 00:01:33.280 Doesn’t matter. After that, you can specialize  it on the domain that you are looking for, and   00:01:33.280 --> 00:01:40.160 when you want to do that—to make it really fast,  this particular process—it's best if you go for   00:01:40.160 --> 00:01:45.520 synthetic data. If you have a way to, actually,  generate very high-quality synthetic data,   00:01:45.520 --> 00:01:52.520 you can fast-track this part of specialization  process. Not only single model. So this year,   00:01:52.520 --> 00:01:56.640 you're going to see a lot more multi-agent  models. And when you are trying to build these   00:01:56.640 --> 00:02:02.360 multi-agent models, you're fearing like, OK, it  might increase the cost too much, the latency too   00:02:02.360 --> 00:02:06.880 much. So it's also very much important that you  have a multi-agent system and you can, sort of,   00:02:06.880 --> 00:02:12.280 replace some of those agents with specialized  small models. And when you're trying to   00:02:12.280 --> 00:02:18.880 address these goals, you want this process to be  something which you know works fast. So that's why   00:02:18.880 --> 00:02:23.740 we are trying to make sure we have a very good way  to create synthetic data for your specific need. 00:02:23.740 --> 00:02:30.240 HUIZINGA: No research exists in a vacuum, and  most of it fills some kind of a gap. So tell us   00:02:30.240 --> 00:02:34.320 what's already been done in this field  and how this work is building on it. 00:02:34.320 --> 00:02:41.520 MITRA: So previously, actually, we have seen  that in post-training, the more data you have,   00:02:41.520 --> 00:02:47.440 the better the performance goes for the model  you're training. So what we wanted to test is how   00:02:47.440 --> 00:02:52.720 much we can scale and what happens if we scale a  lot and lot. But we didn't have the tools for it.   00:02:52.720 --> 00:02:58.480 So the other approaches people previously used  was you had a small set of data and how do we   00:02:58.480 --> 00:03:03.040 expand this dataset into much larger and larger  amount of data. That's where people were mostly   00:03:03.040 --> 00:03:07.840 focusing. But it's not that easy to create that  initial seed set. [LAUGHTER] You need to be very   00:03:07.840 --> 00:03:13.520 expert. The way that we're doing is, actually,  rather you define what you want to create. Like,   00:03:13.520 --> 00:03:17.360 OK, you want to create tool-use data.  So you say, OK, I have a bunch of tools,   00:03:17.360 --> 00:03:22.360 and I am looking for data in the scenarios where  someone can just come give me a description and   00:03:22.360 --> 00:03:27.600 then maybe that person interact with the AI to  figure out how to get the job done. It's not a   00:03:27.600 --> 00:03:32.200 one-step thing. And maybe you also have a setting  where it's more like an app developer. You have   00:03:32.200 --> 00:03:36.880 a bunch of APIs in your phone. You just want to  figure out which one is best for the user request,   00:03:36.880 --> 00:03:39.640 which came through voice command. So  different scenarios could be there. So   00:03:39.640 --> 00:03:43.560 what we're saying [is], OK, we are not going  through the method where you have to come up   00:03:43.560 --> 00:03:47.240 with your initial own seed data and then we  expand. It is more like you define what you   00:03:47.240 --> 00:03:52.280 want to do. It's much more abstract. And then,  we are, sort of, automating the effort of data   00:03:52.280 --> 00:03:56.840 creation. So this setting actually of synthetic  data creation, we are referring [to] it as   00:03:56.840 --> 00:03:59.440 generative teaching, and that's where we  are, sort of, differing. So previously,   00:03:59.440 --> 00:04:05.480 it was more like expansion, and now we are trying  from specification to the data that you need. 00:04:05.480 --> 00:04:08.280 HUIZINGA: Gotcha. Well talk a little bit more   00:04:08.280 --> 00:04:12.600 about your methodology and how you  went about conducting this research. 00:04:12.600 --> 00:04:18.160 MITRA: So first of all, what we are proposing  actually is a multi-agent solution. So you start   00:04:18.160 --> 00:04:22.600 with first describing what you really  need. So you describe in detail, like,   00:04:22.600 --> 00:04:28.440 I need data for this specific skill or this  specific scenario. Then, what we do is like,   00:04:28.440 --> 00:04:34.480 OK, you have some unstructured data or raw data  like text documents or code files that you gather   00:04:34.480 --> 00:04:41.120 from web with permissible license or use something  that you own. We don't care much about what the   00:04:41.120 --> 00:04:46.960 content is really. So it's more like we got some  random stuff, some random content. And then we'll   00:04:46.960 --> 00:04:51.200 guide you how to convert this random something  which is not meaningful for you into something   00:04:51.200 --> 00:04:54.360 which is meaningful for your data creation.  For example, like, if you are creating data   00:04:54.360 --> 00:04:59.200 to teach how to use APIs, you might think about,  you need lots of APIs and how do you get these   00:04:59.200 --> 00:05:04.640 APIs. So what we are saying is, like, we can take  something like code and we'll have agents which   00:05:04.640 --> 00:05:12.760 will convert these raw code files into list of  APIs which is more like a library. So you create   00:05:12.760 --> 00:05:16.800 automatically this input that is very meaningful  for data creation. And then once we have that,   00:05:16.800 --> 00:05:20.560 we have basically the seed instruction creation  step based on your specification. Like, what   00:05:20.560 --> 00:05:24.920 do you want to create data for? So you have all  these different scenarios, and we have multiple   00:05:24.920 --> 00:05:28.360 agents creating data for different scenarios.  And then the last step is actually what we   00:05:28.360 --> 00:05:33.840 call refinement step. So it's more like whatever  data you created, we’ll go through them and we’ll   00:05:33.840 --> 00:05:39.040 make them better and better—improve the quality,  improve the complexity, improve the trickiness,   00:05:39.040 --> 00:05:45.040 we’ll teach when not to answer, etc., etc.  So make sure we cover the whole space. So by   00:05:45.040 --> 00:05:48.973 changing the stochastic seed, we are trying  to cover the entire possible data space. 00:05:48.973 --> 00:05:49.593 HUIZINGA: Right. 00:05:49.593 --> 00:05:54.600 MITRA: So that's the key thing. The way we, sort  of, conducted this research is actually we defined   00:05:54.600 --> 00:06:00.480 17 skills. Skills meaning reading comprehension,  tool use, text modification, content creation,   00:06:00.480 --> 00:06:04.120 RAG (retrieval-augmented generation) ... we have,  like, list of 17 skills … conversation … and then   00:06:04.120 --> 00:06:10.600 we created one multi-agent flow for each of the  skills and we generate data. So one key thing I   00:06:10.600 --> 00:06:16.160 want to highlight is, like, this work, compared  to other work, it was not benchmark driven. We   00:06:16.160 --> 00:06:20.480 want to teach a skill. We don't care which  benchmarks we're trying to evaluate it on.   00:06:20.480 --> 00:06:25.800 So we define the skill, like tool use means this  to us, reading comprehension means this to us,   00:06:25.800 --> 00:06:30.480 text modification means this to us. And then we,  sort of, generate the data to teach everything for   00:06:30.480 --> 00:06:35.800 that skill. And then what we did, we created  actually 22 million instructions. And we had   00:06:35.800 --> 00:06:40.320 previously in Orca series, we had 3 million,  around, instructions. So the 25 million is what   00:06:40.320 --> 00:06:46.280 we, sort of, have at the end. And that's where  we actually trained a Mistral model as of now.   00:06:46.280 --> 00:06:50.400 And we're going to measure, like, how much we  improve the Mistral model by this post-training. 00:06:50.400 --> 00:06:53.440 HUIZINGA: Moving from methods to findings,   00:06:53.440 --> 00:06:57.040 I always look forward to the part of the  research paper that finishes the sentence   00:06:57.040 --> 00:07:02.500 “and what we found was … ,” so give us a quick  overview of your results. What did you find? 00:07:02.500 --> 00:07:09.280 MITRA: Yes, so the results were actually very  exciting for us. So Mistral 7B was our main,   00:07:09.280 --> 00:07:12.240 sort of, baseline because that's  where we’re trying to showcase, like,   00:07:12.240 --> 00:07:16.000 how much improvement we are getting. On the other  side, we have, like, frontier models—ChatGPT,   00:07:16.000 --> 00:07:20.320 GPT-4. We want to also measure how far we  are from those frontier models, so that's,   00:07:20.320 --> 00:07:25.720 sort of, our evaluation setup. So on average  actually, we got like 20 percent performance   00:07:25.720 --> 00:07:31.520 gain over the Mistral, and we evaluated that  across 14 benchmarks that test reasoning,   00:07:31.520 --> 00:07:37.560 content creation, instruction following, format  following, etc. But what was more important to us   00:07:37.560 --> 00:07:41.720 was to do a skill-specific evaluation because we  are trying to teach certain skills, and we had,   00:07:41.720 --> 00:07:45.920 like, 17 skills as we mentioned earlier. So, for  example, like, if you are focusing on reading   00:07:45.920 --> 00:07:51.480 comprehension as a skill, we took LSAT, SAT,  and DROP, and many other benchmarks; we created   00:07:51.480 --> 00:07:55.840 a collection of reading comprehension-based  benchmark. And there, we are observing, like,   00:07:55.840 --> 00:08:00.720 20 percent improvement over Mistral, and what it  means, like, we're actually achieving GPT-4–level   00:08:00.720 --> 00:08:06.600 performance. Similarly, if I'm focusing on math  skill, there are many datasets which test, like,   00:08:06.600 --> 00:08:12.680 elementary math, high school math, college-level  math. And we improved actually across all these   00:08:12.680 --> 00:08:19.080 different levels of math. So we see from  40 percent to 150 percent of improvement   00:08:19.080 --> 00:08:24.440 on different benchmarks of math. So it was  more like what we wanted to see. We're not   00:08:24.440 --> 00:08:27.720 optimizing for a particular benchmark. We  wanted to optimize the skill, and that's   00:08:27.720 --> 00:08:31.600 what you're observing. So you're observing  improvement in math across all these levels,   00:08:31.600 --> 00:08:37.040 from elementary to high school to college to  middle school, etc., everything. The same goes   00:08:37.040 --> 00:08:43.800 for RAG, as well. We’re observing on RAG skill  92 percent, around, improvement over Mistral. The   00:08:43.800 --> 00:08:47.960 format following numbers are pretty interesting  to us. So format following is very important for   00:08:47.960 --> 00:08:52.480 SLMs (small language models). You want to make  these models practical. You want to make sure   00:08:52.480 --> 00:08:57.080 that they follow the format so you can parse  the result. And we were able to take Mistral   00:08:57.080 --> 00:09:02.320 beyond Gemini Pro. So that was a very strong  performance from the post-training that we   00:09:02.320 --> 00:09:07.680 did. For summarization, actually we were able to  reduce the hallucination rate by 31 percent while   00:09:07.680 --> 00:09:12.080 achieving the GPT-4–level quality. So overall,  all these results were, sort of, highlighting   00:09:12.080 --> 00:09:17.060 that the methodology that we have, which we're  calling AgentInstruct, is very promising. 00:09:17.060 --> 00:09:21.400 HUIZINGA: I think it's important to  get practical and talk about real-world   00:09:21.400 --> 00:09:27.480 impact. So tell us who you think this  research will benefit most and why. 00:09:27.480 --> 00:09:34.480 MITRA: Yeah, so again the model builders  will, sort of, find it most beneficial. So the   00:09:34.480 --> 00:09:41.120 significance of our work actually lies in the way  we are trying to revolutionize the language model   00:09:41.120 --> 00:09:46.520 development through scalable, low-effort synthetic  creation. And the scalable and low effort is,   00:09:46.520 --> 00:09:52.200 sort of, the key thing, right. We have shown  that we can create very high-quality data.   00:09:52.200 --> 00:09:56.080 That's what the numbers are telling us. We  want to mention that this is very scalable   00:09:56.080 --> 00:10:01.020 and low effort, and that's what we think  might help the most for model builders. 00:10:01.020 --> 00:10:06.120 HUIZINGA: So, Arindam, let's borrow a phrase  from the machine learning lexicon and go for   00:10:06.120 --> 00:10:11.320 a little one-shot learning here: if you had  to boil down why your work is important,   00:10:11.320 --> 00:10:15.960 what's the one thing you want our  listeners to take away from this research? 00:10:15.960 --> 00:10:20.960 MITRA: The key takeaway would be, like, the  AgentInstruct method enables the generation   00:10:20.960 --> 00:10:26.800 of vast, diverse, and high-quality  synthetic data with very minimal human   00:10:26.800 --> 00:10:30.580 input. So that's one thing I would,  like, to remember from this paper. 00:10:30.580 --> 00:10:36.440 HUIZINGA: So as we close, talk briefly about  the limitations that you encountered in this   00:10:36.440 --> 00:10:41.560 project and directions for future research. What  are the outstanding challenges in this field,   00:10:41.560 --> 00:10:44.460 and what's on your research  agenda to overcome them? 00:10:44.460 --> 00:10:51.080 MITRA: Yes, so we're exploring further automation.  But apart from making this data creation more   00:10:51.080 --> 00:10:57.200 automated and less human involvement needed,  we're trying to focus on two other aspects. One   00:10:57.200 --> 00:11:03.120 is automated model debugging, and the other is  automated model repairing. So now that we have   00:11:03.120 --> 00:11:07.840 the ability to generate data for a particular  skill, let's say math, for model debugging,   00:11:07.840 --> 00:11:14.080 what we need is basically an error handler.  Like something we can plug in which takes   00:11:14.080 --> 00:11:18.240 the question and the answer coming from a  different model and verifies if the answer   00:11:18.240 --> 00:11:23.880 is correct or not. So that's the part we're  working on right now, figuring out this error   00:11:23.880 --> 00:11:29.040 handler. And the second aspect is repairing.  So once we have the error, we figure out, OK,   00:11:29.040 --> 00:11:34.040 this is where the model is struggling. How can we  give feedback or how can we give more knowledge so   00:11:34.040 --> 00:11:38.229 it can basically correct those errors? So those  are some things we're working on right now. 00:11:38.229 --> 00:11:39.550 [MUSIC PLAYS] 00:11:39.550 --> 00:11:43.160 HUIZINGA: Well, Arindam Mitra, thanks for  joining us today, and to our listeners,   00:11:43.160 --> 00:11:51.200 thanks for tuning in. If you want to read this  paper, you can find a link at aka.ms/abstracts,   00:11:51.200 --> 00:11:56.059 or you can find a preprint on arXiv.  See you next time on Abstracts! 00:12:03.957 --> 00:12:10.800 [MUSIC FADES]