WEBVTT Kind: captions Language: en-US 00:00:02.405 --> 00:00:03.107 [MUSIC] 00:00:03.107 --> 00:00:06.800 KATHLEEN SULLIVAN: Welcome  to AI Testing and Evaluation:   00:00:06.800 --> 00:00:11.520 Learnings from Science and Industry.  I'm your host, Kathleen Sullivan. 00:00:11.520 --> 00:00:16.640 As generative AI continues to advance, Microsoft  has gathered a range of experts—from genome   00:00:16.640 --> 00:00:21.040 editing to cybersecurity—to share how  their fields approach evaluation and   00:00:21.040 --> 00:00:26.160 risk assessment. Our goal is to learn from  their successes and their stumbles to move   00:00:26.160 --> 00:00:31.360 the science and practice of AI testing  forward. In this series, we'll explore   00:00:31.360 --> 00:00:40.137 how these insights might help guide the future of  AI development, deployment, and responsible use. 00:00:40.126 --> 00:00:40.659 [MUSIC ENDS] 00:00:40.659 --> 00:00:42.800 SULLIVAN: Today, I'm excited to  welcome Dan Carpenter and Timo   00:00:42.800 --> 00:00:46.400 Minssen to the podcast to explore  testing and risk assessment in the   00:00:46.400 --> 00:00:50.160 areas of pharmaceuticals and  medical devices, respectively. 00:00:50.160 --> 00:00:54.640 Dan Carpenter is chair of the Department  of Government at Harvard University. His   00:00:54.640 --> 00:01:00.160 research spans the sphere of social and political  science, from petitioning in democratic society   00:01:00.160 --> 00:01:05.600 to regulation and government organizations.  His recent work includes the FDA Project,   00:01:05.600 --> 00:01:09.440 which examines pharmaceutical  regulation in the United States. 00:01:09.440 --> 00:01:12.720 Timo is a professor of law at  the University of Copenhagen,   00:01:12.720 --> 00:01:18.400 where he is also director of the Center for  Advanced Studies in Bioscience Innovation Law.   00:01:18.400 --> 00:01:22.080 He specializes in legal aspects of  biomedical innovation, including   00:01:22.080 --> 00:01:27.680 intellectual property law and regulatory law.  He's exercised his expertise as an advisor   00:01:27.680 --> 00:01:33.040 to such organizations as the World Health  Organization and the European Commission. 00:01:33.040 --> 00:01:38.000 And after our conversations, we'll talk to  Microsoft's Chad Atalla, an applied scientist   00:01:38.000 --> 00:01:43.520 in responsible AI, about how we should think  about these insights in the context of AI. 00:01:43.520 --> 00:01:46.160 Daniel, it's a pleasure to  welcome you to the podcast.   00:01:46.160 --> 00:01:49.662 I'm just so appreciative of you being  here. Thanks for joining us today. 00:01:49.662 --> 00:01:50.720 DANIEL CARPENTER: Thanks for having me. 00:01:50.720 --> 00:01:56.480 SULLIVAN: Dan, before we dissect policy,  let's rewind the tape to your origin story.   00:01:56.480 --> 00:02:00.960 Can you take us to the moment that you first  became fascinated with regulators rather than,   00:02:00.960 --> 00:02:05.840 say, politicians? Was there a spark  that pulled you toward the FDA story? 00:02:05.840 --> 00:02:08.720 CARPENTER: At one point during graduate school,   00:02:08.720 --> 00:02:16.960 I was studying a combination of American  politics and political theory, and I did a   00:02:16.960 --> 00:02:24.720 summer interning at the Department of Housing  and Urban Development. And I began to think,   00:02:24.720 --> 00:02:30.960 why don't people study these administrators  more and the rules they make, the, you know,   00:02:30.960 --> 00:02:37.360 inefficiencies, the efficiencies? Really more  from, kind of, a descriptive standpoint, less from   00:02:37.360 --> 00:02:45.520 a normative standpoint. And I was reading a lot  that summer about the Food and Drug Administration   00:02:45.520 --> 00:02:51.590 and some of the decisions it was making on  AIDS drugs. That was a, sort of, a major, … 00:02:51.590 --> 00:02:52.800 SULLIVAN: Right. 00:02:52.800 --> 00:02:57.680 CARPENTER: … sort of, you know, moment in the  news, in the global news as well as the national   00:02:57.680 --> 00:03:06.880 news during, I would say, what? The late ’80s,  early ’90s? And so I began to look into that. 00:03:06.880 --> 00:03:09.520 SULLIVAN: So now that we know what pulled you in,   00:03:09.520 --> 00:03:13.120 let’s zoom out for our listeners.  Give us the whirlwind tour. I think   00:03:13.120 --> 00:03:17.480 most of us know pharma involves years of  trials, but what’s the part we don’t know? 00:03:17.480 --> 00:03:24.480 CARPENTER: So I think when most businesses develop  a product, they all go through some phases of   00:03:24.480 --> 00:03:30.320 research and development and testing. And  I think what's different about the FDA is,   00:03:30.960 --> 00:03:32.880 sort of, two- or three-fold. 00:03:32.880 --> 00:03:38.560 First, a lot of those tests are much more  stringently specified and regulated by the   00:03:38.560 --> 00:03:46.960 government, and second, one of the reasons  for that is that the FDA imposes not simply   00:03:46.960 --> 00:03:56.000 safety requirements upon drugs in particular but  also efficacy requirements. The FDA wants you to   00:03:56.000 --> 00:04:02.720 prove not simply that it's safe and non-toxic but  also that it's effective. And the final thing,   00:04:02.720 --> 00:04:07.280 I think, that makes the FDA different is  that it stands as what I would call the “veto   00:04:07.280 --> 00:04:13.200 player” over R&D [research and development]  to the marketplace. The FDA basically has,   00:04:13.200 --> 00:04:16.880 sort of, this control over  entry to the marketplace. 00:04:16.880 --> 00:04:23.600 And so what that involves is usually first,  a set of human trials where people who have   00:04:23.600 --> 00:04:29.120 no disease take it. And you're only looking  for toxicity generally. Then there's a set   00:04:29.120 --> 00:04:34.160 of Phase 2 trials, where they look more  at safety and a little bit at efficacy,   00:04:34.160 --> 00:04:41.440 and you're now examining people who have the  disease that the drug claims to treat. And   00:04:41.440 --> 00:04:47.440 you're also basically comparing people who  get the drug, often with those who do not. 00:04:47.440 --> 00:04:53.520 And then finally, Phase 3 involves a much more  direct and large-scale attack, if you will,   00:04:53.520 --> 00:04:59.760 or assessment of efficacy, and that's where you  get the sort of large randomized clinical trials   00:04:59.760 --> 00:05:08.000 that are very expensive for pharmaceutical  companies, biomedical companies to launch,   00:05:08.000 --> 00:05:16.640 to execute, to analyze. And those are often the  sort of core evidence base for the decisions that   00:05:16.640 --> 00:05:22.240 the FDA makes about whether or not to approve  a new drug for marketing in the United States. 00:05:22.240 --> 00:05:27.600 SULLIVAN: Are there differences  in how that process has, you know,   00:05:27.600 --> 00:05:32.400 changed through other countries and maybe just  how that's evolved as you've seen it play out? 00:05:32.400 --> 00:05:38.320 CARPENTER: Yeah, for a long time, I would say that  the United States had probably the most stringent   00:05:38.320 --> 00:05:44.960 regime of regulation for biopharmaceutical  products until, I would say, about the 1990s   00:05:44.960 --> 00:05:52.080 and early 2000s. It used to be the case that a  number of other countries, especially in Europe   00:05:52.080 --> 00:05:58.720 but around the world, basically waited for the  FDA to mandate tests on a drug and only after   00:05:58.720 --> 00:06:05.760 the drug was approved in the United States would  they deem it approvable and marketable in their   00:06:05.760 --> 00:06:12.880 own countries. And then after the formation of the  European Union and the creation of the European   00:06:12.880 --> 00:06:20.080 Medicines Agency, gradually the European Medicines  Agency began to get a bit more stringent. 00:06:20.080 --> 00:06:24.720 But, you know, over the long run, there's  been a lot of, sort of, heterogeneity,   00:06:24.720 --> 00:06:29.760 a lot of variation over time and space,  in the way that the FDA has approached   00:06:29.760 --> 00:06:35.920 these problems. And I'd say in the last 20  years, it's begun to partially deregulate,   00:06:37.200 --> 00:06:45.440 namely, you know, trying to find all sorts of  mechanisms or pathways for really innovative   00:06:45.440 --> 00:06:52.800 drugs for deadly diseases without a lot  of treatments to basically get through the   00:06:52.800 --> 00:06:58.880 process at lower cost. For many people, that  has not been sufficient. They're concerned   00:06:58.880 --> 00:07:06.400 about the cost of the system. Of course, then  the agency also gets criticized by those who   00:07:06.400 --> 00:07:13.080 believe it's too lax. It is potentially letting  ineffective and unsafe therapies on the market. 00:07:13.080 --> 00:07:16.960 SULLIVAN: In your view, when does the  structured model genuinely safeguard   00:07:16.960 --> 00:07:21.200 patients and where do you think it  maybe slows or limits innovation? 00:07:21.200 --> 00:07:30.320 CARPENTER: So I think the worry is that if you  approach pharmaceutical approval as a world   00:07:30.320 --> 00:07:36.480 where only things can go wrong, then you're  really at a risk of limiting innovation. And   00:07:36.480 --> 00:07:44.000 even if you end up letting a lot of things  through, if by your regulations you end up   00:07:44.000 --> 00:07:48.480 basically slowing down the development  process or making it very, very costly,   00:07:48.480 --> 00:07:54.000 then there's just a whole bunch of drugs that  either come to market too slowly or they come   00:07:54.000 --> 00:08:00.000 to market not at all because they just aren't  worth the kind of cost-benefit or, sort of,   00:08:00.000 --> 00:08:05.760 profit analysis of the firm. You know, so that's  been a concern. And I think it's been one of the   00:08:05.760 --> 00:08:11.120 reasons that the Food and Drug Administration  as well as other world regulators have begun to   00:08:11.120 --> 00:08:18.640 basically try to smooth the process and  accelerate the process at the margins. 00:08:18.640 --> 00:08:24.000 The other thing is that they've started to  basically make approvals on the basis of what   00:08:24.000 --> 00:08:29.600 are called surrogate endpoints. So the idea  is that a cancer drug, we really want to know   00:08:29.600 --> 00:08:36.720 whether that drug saves lives, but if we wait  to see whose lives are saved or prolonged by   00:08:36.720 --> 00:08:41.360 that drug, we might miss the opportunity  to make judgments on the basis of, well,   00:08:41.360 --> 00:08:47.360 are we detecting tumors in the bloodstream? Or  can we measure the size of those tumors in, say,   00:08:47.360 --> 00:08:53.920 a solid cancer? And then the further question  is, is the size of the tumor basically a really   00:08:53.920 --> 00:08:59.040 good correlate or predictor of whether  people will die or not, right? Generally,   00:08:59.040 --> 00:09:07.760 the FDA tends to be less stringent when you've  got, you know, a remarkably innovative new therapy   00:09:07.760 --> 00:09:14.400 and the disease being treated is one that just  doesn't have a lot of available treatments, right. 00:09:14.960 --> 00:09:18.880 The one thing that people often think  about when they're thinking about   00:09:18.880 --> 00:09:23.510 pharmaceutical regulation is they often  contrast, kind of, speed versus safety, … 00:09:23.510 --> 00:09:24.065 SULLIVAN: Right. 00:09:24.065 --> 00:09:28.400 CARPENTER: … right. And that's useful as a  tradeoff, but I often try to remind people   00:09:28.400 --> 00:09:34.400 that it's not simply about whether the drug  gets out there and it's unsafe. You know,   00:09:34.400 --> 00:09:39.520 you and I as patients and even doctors have  a hard time knowing whether something works   00:09:39.520 --> 00:09:44.960 and whether it should be prescribed.  And the evidence for knowing whether   00:09:44.960 --> 00:09:49.440 something works isn't just, well, you  know, Sally took it, or Dan took it,   00:09:49.440 --> 00:09:54.000 or Kathleen took it, and they seem to get  better or they didn't seem to get better. 00:09:54.000 --> 00:09:58.240 The really rigorous evidence comes from  randomized clinical trials. And I think   00:09:58.240 --> 00:10:03.280 it's fair to say that if you didn't  have the FDA there as a veto player,   00:10:03.280 --> 00:10:07.840 you wouldn't get as many randomized clinical  trials and the evidence probably wouldn't be   00:10:07.840 --> 00:10:12.800 as rigorous for whether these things work. And  as I like to put it, basically there's a whole   00:10:12.800 --> 00:10:18.800 ecology of expectations and beliefs around  the biopharmaceutical industry in the United   00:10:18.800 --> 00:10:24.960 States and globally, and to some extent, it's  undergirded by all of these tests that happen. 00:10:24.960 --> 00:10:25.810 SULLIVAN: Right. 00:10:25.810 --> 00:10:31.040 CARPENTER: And in part, that means it's  undergirded by regulation. Would there still be   00:10:31.040 --> 00:10:36.640 a market without regulation? Yes. But it would be  a market in which people had far less information   00:10:36.640 --> 00:10:43.840 in and confidence about the drugs that are being  taken. And so I think it's important to recognize   00:10:43.840 --> 00:10:51.760 that kind of confidence-boosting potential  of, kind of, a scientific regulation base. 00:10:51.760 --> 00:10:55.040 SULLIVAN: Actually, if we could double-click  on that for a minute, I'd love to hear your   00:10:55.040 --> 00:10:59.600 perspective on, testing has been completed;  there's results. Can you walk us through how   00:10:59.600 --> 00:11:05.360 those results actually shape the next steps and  decisions of a particular drug and just, like, how   00:11:05.360 --> 00:11:11.920 regulators actually think about using that data  to influence really what happens next with it? 00:11:11.920 --> 00:11:18.320 CARPENTER: Right. So it's important to  understand that every drug is approved   00:11:18.320 --> 00:11:23.840 for what's called an indication. It  can have a first primary indication,   00:11:23.840 --> 00:11:28.160 which is the main disease that it treats, and  then others can be added as more evidence is   00:11:28.160 --> 00:11:34.640 shown. But a drug is not something that just  kind of exists out there in the ether. It   00:11:34.640 --> 00:11:39.120 has to have the right form of administration.  Maybe it should be injected. Maybe it should be   00:11:39.120 --> 00:11:44.720 ingested. Maybe it should be administered only  at a clinic because it needs to be kind of   00:11:44.720 --> 00:11:50.720 administered in just the right way. As doctors  will tell you, dosage is everything, right. 00:11:50.720 --> 00:11:56.000 And so one of the reasons that you want  those trials is not simply a, you know,   00:11:56.000 --> 00:12:02.560 yes or no answer about whether the drug  works, right. It's not simply if-then. It's   00:12:02.560 --> 00:12:06.320 literally what goes into what you might  call the dose response curve. You know,   00:12:06.320 --> 00:12:11.600 how much of this drug do we need to basically,  you know, get the benefit? At what point does   00:12:11.600 --> 00:12:17.680 that fall off significantly that we can basically  say, we can stop there? All that evidence comes   00:12:17.680 --> 00:12:25.600 from trials. And that's the kind of evidence  that is required on the basis of regulation. 00:12:25.600 --> 00:12:30.080 Because it's not simply a drug  that's approved. It's a drug and   00:12:30.080 --> 00:12:36.000 a frequency of administration. It's a method  of administration. And so the drug isn't just,   00:12:36.000 --> 00:12:39.120 there's something to be taken off the  shelf and popped into your mouth. I mean,   00:12:39.120 --> 00:12:43.360 sometimes that's what happens, but even then,  we want to know what the dosage is, right.   00:12:44.160 --> 00:12:47.760 We want to know what to look for in  terms of side effects, things like that. 00:12:47.760 --> 00:12:49.440 SULLIVAN: Going back to that point, I mean,   00:12:49.440 --> 00:12:55.920 it sounds like we're making a lot of progress  from a regulation perspective in, you know,   00:12:55.920 --> 00:13:01.200 sort of speed and getting things approved but  doing it in a really balanced way. I mean,   00:13:01.200 --> 00:13:05.840 any other kind of closing thoughts on the  tradeoffs there or where you're seeing that going? 00:13:05.840 --> 00:13:10.560 CARPENTER: I think you're going to see some move  in the coming years—there's already been some of   00:13:10.560 --> 00:13:16.400 it—to say, do we always need a really large  Phase 3 clinical trial? And to what degree do   00:13:16.400 --> 00:13:21.040 we need the, like, you know, all the i's dotted  and the t's crossed or a really, really large   00:13:21.040 --> 00:13:29.920 sample size? And I'm open to innovation there.  I'm also open to the idea that we consider, again,   00:13:29.920 --> 00:13:37.520 things like accelerated approvals or pathways  for looking at different kinds of surrogate   00:13:37.520 --> 00:13:42.360 endpoints. I do think, once we do that, then  we also have to have some degree of follow-up. 00:13:42.360 --> 00:13:44.480 SULLIVAN: So I know we're  getting close to out of time,   00:13:44.480 --> 00:13:50.800 but maybe just a quick rapid fire if you’re  open to it. Biggest myth about clinical trials? 00:13:50.800 --> 00:13:54.640 CARPENTER: Well, some people tend to think  that the FDA performs them. You know,   00:13:54.640 --> 00:14:00.160 it's companies that do it. And the only other  thing I would say is the company that does a   00:14:00.160 --> 00:14:05.600 lot of the testing and even the innovating is not  always the company that takes the drug to market,   00:14:05.600 --> 00:14:12.240 and it tells you something about how powerful  regulation is in our system, in our world,   00:14:12.240 --> 00:14:18.640 that you often need a company that has dealt with  the FDA quite a bit and knows all the regulations   00:14:18.640 --> 00:14:23.760 and knows how to dot the i's and cross the t's  in order to get a drug across the finish line. 00:14:23.760 --> 00:14:27.920 SULLIVAN: If you had a magic wand, what's the  one thing you'd change in regulation today? 00:14:27.920 --> 00:14:32.960 CARPENTER: I would like people to think a little  bit less about just speed versus safety and,   00:14:32.960 --> 00:14:37.280 again, more about this basic issue of  confidence. I think it's fundamental   00:14:37.280 --> 00:14:41.200 to everything that happens in markets  but especially in biopharmaceuticals. 00:14:41.200 --> 00:14:45.821 SULLIVAN: Such a great point. This has been really  fun. Just thanks so much for being here today. 00:14:45.821 --> 00:14:46.358 [TRANSITION MUSIC] 00:14:46.358 --> 00:14:49.920 We're really excited to share your  thoughts out to our listeners. Thanks. 00:14:49.920 --> 00:14:54.160 CARPENTER: Likewise. 00:14:58.120 --> 00:14:59.120 SULLIVAN:   00:14:59.120 --> 00:15:04.480 Now to the world of medical devices, I'm joined  by Professor Timo Minssen. Professor Minssen,   00:15:04.480 --> 00:15:07.746 it's great to have you here.  Thank you for joining us today. 00:15:07.746 --> 00:15:09.720 TIMO MINSSEN: Yeah, thank you  very much, it's a pleasure. 00:15:09.720 --> 00:15:13.440 SULLIVAN: Before getting into the  regulatory world of medical devices,   00:15:13.440 --> 00:15:17.440 tell our audience a bit about your personal  journey or your origin story, as we're asking   00:15:17.440 --> 00:15:22.480 our guests. How did you land in regulation,  and what's kept you hooked in this space? 00:15:22.480 --> 00:15:28.160 MINSSEN: So I started out as a patent expert in  the biomedical area, starting with my PhD thesis   00:15:28.160 --> 00:15:34.240 on patenting biologics in Europe and in the US.  So during that time, I was mostly interested in   00:15:34.240 --> 00:15:39.200 patent and trade secret questions. But at the  same time, I also developed and taught courses   00:15:39.200 --> 00:15:45.680 in regulatory law and held talks on regulating  advanced medical therapy medicinal products.   00:15:45.680 --> 00:15:52.720 I then started to lead large research projects  on legal challenges in a wide variety of health   00:15:52.720 --> 00:15:57.840 and life science innovation frontiers. I also  started to focus increasingly on AI-enabled   00:15:57.840 --> 00:16:03.760 medical devices and software as a medical device,  resulting in several academic articles in this   00:16:03.760 --> 00:16:09.800 area and also in the regulatory area and a book  on the future of medical device regulation. 00:16:09.800 --> 00:16:12.800 SULLIVAN: Yeah, what's kept  you hooked in the space? 00:16:12.800 --> 00:16:15.600 MINSSEN: It's just incredibly exciting,   00:16:15.600 --> 00:16:22.880 in particular right now with everything that  is going on, you know, in the software arena,   00:16:22.880 --> 00:16:28.000 in the marriage between AI and medical  devices. And this is really challenging   00:16:28.000 --> 00:16:32.360 not only societies but also regulators  and authorities in Europe and in the US. 00:16:32.360 --> 00:16:35.760 SULLIVAN: Yeah, it's a super exciting  time to be in this space. You know,   00:16:35.760 --> 00:16:39.360 we talked to Daniel a little earlier and,  you know, I think similar to pharmaceuticals,   00:16:39.360 --> 00:16:45.280 people have a general sense of what we mean when  we say medical devices, but most listeners may   00:16:45.280 --> 00:16:52.240 picture like a stethoscope or a hip implant.  The word “medical device” reaches much wider.   00:16:52.240 --> 00:16:57.120 Can you give us a quick, kind of, range from  perhaps very simple to even, I don't know,   00:16:57.120 --> 00:17:03.040 sci-fi and then your 90-second tour of how risk  assessment works and why a framework is essential? 00:17:03.040 --> 00:17:06.800 MINSSEN: Let me start out by saying that the WHO  [World Health Organization] estimates that today   00:17:06.800 --> 00:17:11.840 there are approximately 2 million different  kinds of medical devices on the world market,   00:17:11.840 --> 00:17:19.360 and as of the FDA's latest update that I'm  aware of, the FDA has authorized more than 1,000   00:17:19.360 --> 00:17:24.720 AI-, machine learning-enabled medical  devices, and that number is rising rapidly. 00:17:24.720 --> 00:17:29.040 So in that context, I think it is important  to understand that medical devices can be   00:17:29.040 --> 00:17:35.520 any instrument, apparatus, implement, machine,  appliance, implant, reagent for in vitro use,   00:17:35.520 --> 00:17:42.080 software, material, or other similar or related  articles that are intended by the manufacturer to   00:17:42.080 --> 00:17:47.920 be used alone or in combination for a medical  purpose. And the spectrum of what constitutes   00:17:47.920 --> 00:17:52.800 a medical device can thus range from very  simple devices such as tongue depressors,   00:17:52.800 --> 00:17:58.240 contact lenses, and thermometers to more  complex devices such as blood pressure monitors,   00:17:58.240 --> 00:18:04.960 insulin pumps, MRI machines, implantable  pacemakers, and even software as a medical   00:18:04.960 --> 00:18:09.680 device or AI-enabled monitors or  drug device combinations, as well. 00:18:09.680 --> 00:18:13.520 So talking about regulation, I think  it is also very important to stress   00:18:13.520 --> 00:18:17.280 that medical devices are used in  many diverse situations by very   00:18:17.280 --> 00:18:22.480 different stakeholders. And testing has  to take this variety into consideration,   00:18:22.480 --> 00:18:28.560 and it is intrinsically tied to regulatory  requirements across various jurisdictions. 00:18:28.560 --> 00:18:33.120 During the pre-market phase, medical  testing establishes baseline safety   00:18:33.120 --> 00:18:37.120 and effectiveness metrics through  bench testing, performance standards,   00:18:37.120 --> 00:18:43.360 and clinical studies. And post-market testing  ensures that real-world data informs ongoing   00:18:43.360 --> 00:18:49.120 compliance and safety improvements. So testing  is indispensable in translating technological   00:18:49.120 --> 00:18:53.600 innovation into safe and effective  medical devices. And while particular   00:18:53.600 --> 00:18:58.960 details of pre-market and post-market review  procedures may slightly differ among countries,   00:18:58.960 --> 00:19:04.640 most developed jurisdictions regulate medical  devices similarly to the US or European models.   00:19:04.640 --> 00:19:09.840 So most jurisdictions with medical device  regulation classify devices based on their   00:19:09.840 --> 00:19:15.440 risk profile, intended use, indications  for use, technological characteristics,   00:19:15.440 --> 00:19:22.200 and the regulatory controls necessary to provide a  reasonable assurance of safety and effectiveness. 00:19:22.200 --> 00:19:27.280 SULLIVAN: So medical devices face  a pretty prescriptive multi-level   00:19:27.280 --> 00:19:30.320 testing path before they hit the  market. From your vantage point,   00:19:30.320 --> 00:19:34.120 what are some of the downsides of that  system and when does it make the most sense? 00:19:34.120 --> 00:19:39.440 MINSSEN: One primary drawback is, of course,  the lengthy and expensive approval process.   00:19:39.440 --> 00:19:44.240 High-risk devices, for example, often  undergo years of clinical trials,   00:19:44.240 --> 00:19:47.840 which can cost millions of dollars, and  this can create a significant barrier   00:19:47.840 --> 00:19:54.240 for startups and small companies with limited  resources. And even for moderate-risk devices,   00:19:54.240 --> 00:19:59.520 the regulatory burden can slow product  development and time to the market. 00:19:59.520 --> 00:20:05.760 And the approach can also limit flexibility.  Prescriptive requirements may not accommodate   00:20:05.760 --> 00:20:10.800 emerging innovations like digital therapeutics  or AI-based diagnostics in a feasible way. And   00:20:10.800 --> 00:20:16.240 in such cases, the framework can unintentionally  [stiffen] innovation by discouraging creative   00:20:16.240 --> 00:20:20.960 solutions or iterative improvements,  which as matter of fact can also put   00:20:20.960 --> 00:20:27.040 patients at risk when you don't use new  technologies and AI. And additionally,   00:20:27.040 --> 00:20:31.680 the same level of scrutiny may be  applied to low-risk devices, where   00:20:31.680 --> 00:20:37.680 the extensive testing and documentation may also  be disproportionate to the actual patient risk. 00:20:37.680 --> 00:20:42.320 However, the prescriptive model is  highly appropriate where we have high   00:20:42.320 --> 00:20:47.360 testing standards for high-risk medical  devices, in my view, particularly those   00:20:47.360 --> 00:20:52.960 that are life-sustaining, implanted,  or involve new materials or mechanisms. 00:20:52.960 --> 00:20:57.680 I also wanted to say that I think that  these higher compliance thresholds can   00:20:57.680 --> 00:21:03.520 be OK and necessary if you have a system  where authorities and stakeholders also   00:21:03.520 --> 00:21:10.320 have the capacity and funding to enforce,  monitor, and achieve compliance with such   00:21:10.320 --> 00:21:14.240 rules in a feasible, time-effective,  and straightforward manner. And this,   00:21:14.240 --> 00:21:18.480 of course, requires resources,  novel solutions, and investments. 00:21:18.480 --> 00:21:23.600 SULLIVAN: A range of tests are undertaken across  the life cycle of medical devices. How do these   00:21:23.600 --> 00:21:28.480 testing requirements vary across different stages  of development and across various applications? 00:21:28.480 --> 00:21:35.280 MINSSEN: Yes, that's a good question. So  I think first it is important to realize   00:21:35.280 --> 00:21:42.160 that testing is conducted by various entities,  including manufacturers, independent third-party   00:21:42.160 --> 00:21:49.120 laboratories, and regulatory agencies. And  it occurs throughout the device life cycle,   00:21:49.120 --> 00:21:54.000 beginning with iterative testing during the  research and development stage, advancing   00:21:54.000 --> 00:22:01.120 to pre-market evaluations, and continuing into  post-market monitoring. And the outcomes of these   00:22:01.120 --> 00:22:09.280 tests directly impact regulatory approvals, market  access, and device design refinements, as well.   00:22:09.840 --> 00:22:14.800 So the testing results are typically shared  with regulatory authorities and in some cases   00:22:14.800 --> 00:22:20.480 with healthcare providers and the broader  public to enhance transparency and trust. 00:22:20.480 --> 00:22:25.520 So if you talk about the different phases that  play a role here … so let's turn to the pre-market   00:22:25.520 --> 00:22:31.760 phase, where manufacturers must demonstrate  that the device is conformed to safety and   00:22:31.760 --> 00:22:38.560 performance benchmarks defined by regulatory  authorities. Pre-market evaluations include   00:22:38.560 --> 00:22:45.360 functional bench testing, biocompatibility, for  example, assessments and software validation,   00:22:45.360 --> 00:22:50.960 all of which are integral components  of a manufacturer's submission. 00:22:50.960 --> 00:22:55.520 But, yes, but, testing also, and  we touched already up on that,   00:22:55.520 --> 00:23:03.200 extends into the post-market phase, where it  continues to ensure device safety and efficacy,   00:23:03.200 --> 00:23:09.280 and post-market surveillance relies on testing  to monitor real-world performance and identify   00:23:09.280 --> 00:23:15.200 emerging risks on the post-market phase. By  integrating real-world evidence into ongoing   00:23:15.200 --> 00:23:22.560 assessments, manufacturers can address unforeseen  issues, update devices as needed, and maintain   00:23:22.560 --> 00:23:27.840 compliance with evolving regulatory expectations.  And I think this is particularly important in   00:23:27.840 --> 00:23:33.440 this new generation of medical devices that  are AI-enabled or machine-learning enabled. 00:23:33.440 --> 00:23:38.640 I think we have to understand that in this  AI-enabled medical devices field, you know,   00:23:38.640 --> 00:23:45.040 the devices and the algorithms that are working  with them, they can improve in the lifetime of a   00:23:45.040 --> 00:23:51.200 product. So actually, not only you could assess  them and make sure that they maintain safe,   00:23:51.200 --> 00:23:56.880 you could also sometimes lower the risk  category by finding evidence that these   00:23:56.880 --> 00:24:04.720 devices are actually becoming more precise and  safer. So it can both, you know, heighten the risk   00:24:04.720 --> 00:24:11.520 category or lower the risk category, and that's  why this continuous testing is so important. 00:24:11.520 --> 00:24:15.360 SULLIVAN: Given what you just said, how  should regulators handle a device whose   00:24:15.360 --> 00:24:18.480 algorithm keeps updating itself after approval?  00:24:18.480 --> 00:24:27.040 MINSSEN: Well, it has to be an iterative process  that is feasible and straightforward and that is   00:24:27.040 --> 00:24:33.520 based on a very efficient, both time efficient  and performance efficient, communication between   00:24:33.520 --> 00:24:39.760 the regulatory authorities and the medical device  developers, right. We need to have the sensors in   00:24:39.760 --> 00:24:50.240 place that spot potential changes, and we need  to have the mechanisms in place that allow us to   00:24:50.240 --> 00:24:57.440 quickly react to these changes both regulatory  wise and also in the technological way.  00:24:58.320 --> 00:25:04.880 So I think communication is important, and we need  to have the pathways and the feedback loops in the   00:25:04.880 --> 00:25:14.680 regulation that quickly allow us to monitor  these self-learning algorithms and devices. 00:25:14.680 --> 00:25:19.840 SULLIVAN: It sounds like it's just … there's such  a delicate balance between advancing technology   00:25:19.840 --> 00:25:26.000 and really ensuring public safety. You know, if  we clamp down too hard, we stifle that innovation.   00:25:26.000 --> 00:25:31.760 You already touched upon this a bit. But if we're  too lax, we risk unintended consequences. And I'd   00:25:31.760 --> 00:25:37.200 just love to hear how you think the field is  balancing that and any learnings you can share. 00:25:37.200 --> 00:25:41.920 MINSSEN: So this is very true, and you just  touched upon a very central question also   00:25:41.920 --> 00:25:47.440 in our research and our writing. And this is  also the reason why medical device regulation   00:25:47.440 --> 00:25:53.600 is so fascinating and continues to evolve in  response to rapid advancements in technologies,   00:25:53.600 --> 00:25:58.560 particularly dual technologies regarding  digital health, artificial intelligence,   00:25:58.560 --> 00:26:01.040 for example, and personalized medicine. 00:26:01.040 --> 00:26:05.360 And finding the balance is tricky because  also [a] related major future challenge   00:26:06.160 --> 00:26:10.880 relates to the increasing regulatory  jungle and the complex interplay between   00:26:10.880 --> 00:26:16.640 evolving regulatory landscapes  that regulate AI more generally. 00:26:16.640 --> 00:26:20.960 We really need to make sure that the regulatory  authorities that deal with this, that need to   00:26:20.960 --> 00:26:27.200 find the right balance to promote innovation  and mitigate and prevent risks, need to have   00:26:27.200 --> 00:26:32.560 the capacity to do this. So this requires  investments, and it also requires new ways   00:26:32.560 --> 00:26:39.960 to regulate this technology more flexibly, for  example through regulatory sandboxes and so on. 00:26:39.960 --> 00:26:43.600 SULLIVAN: Could you just expand upon  that a bit and double-click on what   00:26:43.600 --> 00:26:47.360 it is you're seeing there? What excites  you about what's happening in that space? 00:26:47.360 --> 00:26:53.600 MINSSEN: Yes, well, the research of my group at  the Center for Advanced Studies in Bioscience   00:26:53.600 --> 00:27:01.680 Innovation Law is very broad. I mean, we are  looking into gene editing technologies. We are   00:27:01.680 --> 00:27:09.520 looking into new biologics. We are looking  into medical devices, as well, obviously,   00:27:09.520 --> 00:27:12.800 but also other technologies  in advanced medical computing. 00:27:12.800 --> 00:27:20.400 And what we see across the line here is that  there is an increasing demand for having   00:27:20.400 --> 00:27:26.000 more adaptive and flexible regulatory  frameworks in these new technologies,   00:27:26.000 --> 00:27:31.520 in particular when they have new uses,  regulations that are focusing more on   00:27:31.520 --> 00:27:37.520 the product rather than the process. And I  have recently written a report, for example,   00:27:37.520 --> 00:27:44.720 for emerging biotechnologies and bio-solutions  for the EU commission. And even in that area,   00:27:44.720 --> 00:27:50.160 regulatory sandboxes are increasingly  important, increasingly considered. 00:27:50.160 --> 00:27:56.480 So this idea of regulatory sandboxes has been  developing originally in the financial sector,   00:27:56.480 --> 00:28:02.560 and it is now penetrating into other  sectors, including synthetic biology,   00:28:03.360 --> 00:28:08.720 emerging biotechnologies, gene  editing, AI, quantum technology,   00:28:08.720 --> 00:28:15.200 as well. This is basically creating an  environment where actors can test new   00:28:15.200 --> 00:28:21.520 ideas in close collaboration and under  the oversight of regulatory authorities. 00:28:21.520 --> 00:28:29.680 But to implement this in the AI sector now also  leads us to a lot of questions and challenges.   00:28:29.680 --> 00:28:37.280 For example, you need to have the capacities of  authorities that are governing and monitoring and   00:28:37.280 --> 00:28:43.360 deciding on these regulatory sandboxes. There are  issues relating to competition law, for example,   00:28:43.360 --> 00:28:49.680 which you call antitrust law in the US, because  the question is, who can enter the sandbox and how   00:28:49.680 --> 00:28:57.200 may they compete after they exit the sandbox?  And there are many questions relating to,   00:28:57.200 --> 00:29:03.411 how should we work with these sandboxes and  how should we implement these sandboxes? 00:29:03.411 --> 00:29:04.055 [TRANSITION MUSIC] 00:29:04.055 --> 00:29:06.520 SULLIVAN: Well, Timo, it has just been  such a pleasure to speak with you today. 00:29:06.520 --> 00:29:08.560 MINSSEN: Yes, thank you very much. 00:29:16.800 --> 00:29:19.600 And now I'm happy to introduce Chad Atalla. 00:29:19.600 --> 00:29:23.760 Chad is senior applied scientist in  Microsoft Research New York City's   00:29:23.760 --> 00:29:28.800 Sociotechnical Alignment Center, where they  contribute to foundational responsible AI   00:29:28.800 --> 00:29:33.280 research and practical responsible AI  solutions for teams across Microsoft. 00:29:33.280 --> 00:29:34.627 Chad, welcome! 00:29:34.627 --> 00:29:35.200 CHAD ATALLA: Thank you. 00:29:35.200 --> 00:29:40.480 SULLIVAN: So we'll kick off with a couple  questions just to dive right in. So tell me a   00:29:40.480 --> 00:29:46.240 little bit more about the Sociotechnical Alignment  Center, or STAC? I know it was founded in 2022.   00:29:46.240 --> 00:29:50.160 I'd love to just learn a little bit more about  what the group does, how you're thinking about   00:29:50.160 --> 00:29:54.320 evaluating AI, and maybe just give us a sense  of some of the projects you're working on. 00:29:54.320 --> 00:29:57.486 ATALLA: Yeah, absolutely.  The name is quite a mouthful. 00:29:57.486 --> 00:29:58.044 SULLIVAN: It is! [LAUGHS] 00:29:58.044 --> 00:30:01.000 ATALLA: So let's start by breaking  that down and seeing what that means. 00:30:01.000 --> 00:30:01.943 SULLIVAN: Great. 00:30:01.943 --> 00:30:05.360 ATALLA: So modern AI systems  are sociotechnical systems,   00:30:05.360 --> 00:30:10.160 meaning that the social and technical  aspects are deeply intertwined. And   00:30:10.160 --> 00:30:16.560 we're interested in aligning the behaviors of  these sociotechnical systems with some values.   00:30:16.560 --> 00:30:21.520 Those could be societal values; they could  be regulatory values, organizational values,   00:30:21.520 --> 00:30:28.560 etc. And to make this alignment happen, we  need the ability to evaluate the systems. 00:30:28.560 --> 00:30:34.640 So my team is broadly working on an evaluation  framework that acknowledges the sociotechnical   00:30:34.640 --> 00:30:40.640 nature of the technology and the often-abstract  nature of the concepts we're actually interested   00:30:40.640 --> 00:30:46.800 in evaluating. As you noted, it's an applied  science team, so we split our time between some   00:30:46.800 --> 00:30:54.320 fundamental research and time to bridge the  work into real products across the company.   00:30:54.320 --> 00:30:59.760 And I also want to note that to power this  sort of work, we have an interdisciplinary team   00:30:59.760 --> 00:31:04.320 drawing upon the social sciences, linguistics,  statistics, and, of course, computer science. 00:31:04.320 --> 00:31:08.560 SULLIVAN: Well, I'm eager to get into our  takeaways from the conversation with both   00:31:08.560 --> 00:31:12.240 Daniel and Timo. But maybe just to  double-click on this for a minute,   00:31:12.240 --> 00:31:16.960 can you talk a bit about some of the overarching  goals of the AI evaluations that you noted? 00:31:16.960 --> 00:31:25.040 ATALLA: So evaluation is really the act of making  valuative judgments based on some evidence,   00:31:25.040 --> 00:31:32.160 and in the case of AI evaluation, that evidence  might be from tests or measurements, right. And   00:31:32.160 --> 00:31:38.240 the goal of why we're doing this in the first  place is to make decisions and claims most often. 00:31:38.240 --> 00:31:44.080 So perhaps I am going to make a claim about  a model that I'm producing, and I want to say   00:31:44.080 --> 00:31:50.560 that it's better than this other model. Or we are  asking whether a certain product is safe to ship.   00:31:50.560 --> 00:31:57.360 All of these decisions need to be informed by  good evaluation and therefore good measurement   00:31:57.360 --> 00:32:05.040 or testing. And I'll also note that in the  regulatory conversation, risk is often what   00:32:05.040 --> 00:32:11.160 we want to evaluate. So that is a goal in and  of itself. And I'll touch more on that later. 00:32:11.160 --> 00:32:15.280 SULLIVAN: I read a recent paper that you  had put out with some of our colleagues   00:32:15.280 --> 00:32:19.040 from Microsoft Research, from the  University of Michigan, and Stanford,   00:32:19.040 --> 00:32:23.360 and you were arguing that evaluating  generative AI is the social-science   00:32:23.360 --> 00:32:27.120 measurement challenge. Maybe for those  who haven't read the paper, what does this   00:32:27.120 --> 00:32:31.200 mean? And can you tell us a little bit more  about what motivated you and your coauthors? 00:32:31.200 --> 00:32:37.280 ATALLA: So the measurement tasks involved in  evaluating generative AI systems are often   00:32:37.280 --> 00:32:42.400 abstract and contested. So that means  they cannot be directly measured and   00:32:42.400 --> 00:32:48.400 must instead [be] indirectly measured via other  observable phenomena. So this is very different   00:32:48.400 --> 00:32:54.080 than the older machine learning paradigm, where,  let's say, for example, I had a system that took a   00:32:54.080 --> 00:32:59.440 picture of a traffic light and told you whether  it was green, yellow, or red at a given time. 00:32:59.440 --> 00:33:05.760 If we wanted to evaluate that system, the task is  much simpler. But with the modern generative AI   00:33:05.760 --> 00:33:15.200 systems that are also general purpose, they have  open-ended output, and language in a whole chat or   00:33:15.200 --> 00:33:20.480 multiple paragraphs being outputted can have a lot  of different properties. And as I noted, these are   00:33:20.480 --> 00:33:25.920 general-purpose systems, so we don't know exactly  what task they're supposed to be carrying out. 00:33:25.920 --> 00:33:32.640 So then the question becomes, if I want to make  some decision or claim—maybe I want to make a   00:33:32.640 --> 00:33:41.200 claim that this system has human-level reasoning  capabilities—well, what does that mean? Do I have   00:33:41.200 --> 00:33:48.320 the same impression of what that means as you  do? And how do we know whether the downstream,   00:33:48.320 --> 00:33:53.520 you know, measurements and tests that I'm  conducting actually will support my notion of   00:33:53.520 --> 00:33:59.920 what it means to have human-level reasoning,  right? Difficult questions. But luckily,   00:33:59.920 --> 00:34:03.840 social scientists have been dealing with these  exact sorts of challenges for multiple decades   00:34:03.840 --> 00:34:09.920 in fields like education, political science, and  psychometrics. So we're really attempting to avoid   00:34:09.920 --> 00:34:16.000 reinventing the wheel here and trying  to learn from their past methodologies. 00:34:16.000 --> 00:34:21.200 And so the rest of the paper goes on to delve into  a four-level framework, a measurement framework,   00:34:21.200 --> 00:34:26.320 that's grounded in the measurement theory from  the quantitative social sciences that takes us   00:34:26.320 --> 00:34:32.960 all the way from these abstract and contested  concepts through processes to get much clearer   00:34:32.960 --> 00:34:37.320 and eventually reach reliable and valid  measurements that can power our evaluations. 00:34:37.320 --> 00:34:40.400 SULLIVAN: I love that. I mean, that's  the whole point of this podcast, too,   00:34:40.400 --> 00:34:44.320 right. Is to really build on those other  learnings and frameworks that we're taking   00:34:44.320 --> 00:34:49.760 from industries that have been thinking about this  for much longer. Maybe from your vantage point,   00:34:49.760 --> 00:34:54.880 what are some of the biggest day-to-day  hurdles in building solid AI evaluations and,   00:34:54.880 --> 00:34:59.040 I don't know, do we need more shared  standards? Are there bespoke methods?   00:34:59.040 --> 00:35:01.680 Are those the way to go? I would love  to just hear your thoughts on that. 00:35:01.680 --> 00:35:06.800 ATALLA: So let's talk about some of those  practical challenges. And I want to briefly   00:35:06.800 --> 00:35:12.400 go back to what I mentioned about risk before,  all right. Oftentimes, some of the regulatory   00:35:12.400 --> 00:35:18.640 environment is requiring practitioners to measure  the risk involved in deploying one of their   00:35:18.640 --> 00:35:27.200 models or AI systems. Now, risk is importantly  a concept that includes both event and impact,   00:35:27.200 --> 00:35:33.280 right. So there's the probability of some event  occurring. For the case of AI evaluation, perhaps   00:35:33.280 --> 00:35:41.520 this is us seeing a certain AI behavior exhibited.  Then there's also the severity of the impacts,   00:35:41.520 --> 00:35:47.840 and this is a complex chain of effects  in the real world that happen to people,   00:35:47.840 --> 00:35:56.640 organizations, systems, etc., and it's a lot  more challenging to observe the impacts, right. 00:35:56.640 --> 00:36:03.280 So if we're saying that we need to measure  risk, we have to measure both the event and   00:36:03.280 --> 00:36:09.280 the impacts. But realistically, right now,  the field is not doing a very good job of   00:36:09.280 --> 00:36:14.320 actually measuring the impacts. This requires  vastly different techniques and methodologies   00:36:14.320 --> 00:36:19.120 where if I just wanted to measure something  about the event itself, I can, you know,   00:36:19.120 --> 00:36:26.160 do that in a technical sandbox environment and  perhaps have some automated methods to detect   00:36:26.160 --> 00:36:30.720 whether a certain AI behavior is being exhibited.  But if I want to measure the impacts? Now,   00:36:30.720 --> 00:36:35.360 we're in the realm of needing to have real people  involved, and perhaps a longitudinal study where   00:36:35.360 --> 00:36:39.520 you have interviews, questionnaires,  and more qualitative evidence-gathering   00:36:39.520 --> 00:36:47.200 techniques to truly understand the long-term  impacts. So that's a significant challenge. 00:36:47.200 --> 00:36:52.560 Another is that, you know, let's say we forget  about the impacts for now and we focus on the   00:36:52.560 --> 00:37:00.560 event side of things. Still, we need datasets, we  need annotations, and we need metrics to make this   00:37:00.560 --> 00:37:08.000 whole thing work. When I say we need datasets,  if I want to test whether my system has good   00:37:08.000 --> 00:37:14.240 mathematical reasoning, what questions should I  ask? What are my set of inputs that are relevant?   00:37:14.240 --> 00:37:18.800 And then when I get the response from the system,  how do I annotate them? How do I know if it was a   00:37:18.800 --> 00:37:23.840 good response that did demonstrate mathematical  reasoning or if it was a mediocre response?   00:37:24.640 --> 00:37:29.680 And then once I have an annotation of  all of these outputs from the AI system,   00:37:29.680 --> 00:37:33.800 how do I aggregate those all up  into a single informative number? 00:37:33.800 --> 00:37:38.320 SULLIVAN: Earlier in this episode, we  heard Daniel and Timo walk through the   00:37:38.320 --> 00:37:42.960 regulatory frameworks in pharma and medical  devices. I'd be curious what pieces of those   00:37:42.960 --> 00:37:47.240 mature systems are already showing up or at  least may be bubbling up in AI governance. 00:37:47.240 --> 00:37:53.200 ATALLA: Great question. You know, Timo was  talking about the pre-market and post-market   00:37:53.200 --> 00:38:00.720 testing difference. Of course, this is similarly  important in the AI evaluation space. But again,   00:38:00.720 --> 00:38:04.720 these have different methodologies  and serve different purposes. 00:38:04.720 --> 00:38:12.400 So within the pre-deployment phase, we  don't have evidence of how people are   00:38:12.400 --> 00:38:16.560 going to use the system. And when we  have these general-purpose AI systems,   00:38:16.560 --> 00:38:22.320 to understand what the risks are, we really  need to have a sense of what might happen   00:38:22.320 --> 00:38:26.640 and how they might be used. So there are  significant challenges there where I think   00:38:26.640 --> 00:38:33.680 we can learn from other fields and how they do  pre-market testing. And the difference in that   00:38:33.680 --> 00:38:38.800 pre- versus post-market testing also ties to  testing at different stages in the life cycle. 00:38:38.800 --> 00:38:45.920 For AI systems, we already see some regulations  saying you need to start with the base model   00:38:45.920 --> 00:38:52.160 and do some evaluation of the base model,  some basic attributes, some core attributes,   00:38:52.160 --> 00:38:57.760 of that base model before you start putting  it into any real products. But once we have   00:38:57.760 --> 00:39:03.520 a product in mind, we have a user base in mind,  we have a specific task—like maybe we're going   00:39:03.520 --> 00:39:07.440 to integrate this model into Outlook and  it's going to help you write emails—now   00:39:07.440 --> 00:39:12.320 we suddenly have a much crisper picture of  how the system will interact with the world   00:39:12.320 --> 00:39:18.400 around it. And again, at that stage, we need  to think about another round of evaluation. 00:39:18.400 --> 00:39:23.680 Another part that jumped out to me in what  they were saying about pharmaceuticals is   00:39:23.680 --> 00:39:28.080 that sometimes approvals can be  based on surrogate endpoints. So   00:39:28.080 --> 00:39:33.120 this is like we're choosing some heuristic.  Instead of measuring the long-term impact,   00:39:33.120 --> 00:39:36.880 which is what we actually care about,  perhaps we have a proxy that we feel   00:39:36.880 --> 00:39:42.960 like is a good enough indicator of what  that long-term impact might look like. 00:39:42.960 --> 00:39:50.480 This is occurring in the AI evaluation space  right now and is often perhaps even the default   00:39:50.480 --> 00:39:56.320 here since we're not seeing that many studies  of the long-term impact itself. We are seeing,   00:39:56.320 --> 00:40:03.600 instead, folks constructing these heuristics or  proxies and saying if I see this behavior happen,   00:40:03.600 --> 00:40:09.440 I'm going to assume that it indicates this  sort of impact will happen downstream. And   00:40:09.440 --> 00:40:16.480 that's great. It's one of the techniques that  was used to speed up and reduce the barrier   00:40:16.480 --> 00:40:22.320 to innovation in the other fields. And I think  it's great that we are applying that in the AI   00:40:22.320 --> 00:40:27.920 evaluation space. But special care is, of course,  needed to ensure that those heuristics and proxies   00:40:27.920 --> 00:40:33.280 you're using are reasonable indicators of  the greater outcome you're looking for. 00:40:33.280 --> 00:40:38.880 SULLIVAN: What are some of the promising ideas  from maybe pharma or med device regulation   00:40:38.880 --> 00:40:43.360 that maybe haven't made it to AI testing  yet and maybe should? And where would you   00:40:43.360 --> 00:40:47.440 urge technologists, policymakers, and  researchers to focus their energy next? 00:40:47.440 --> 00:40:51.200 ATALLA: Well, one of the key things  that jumped out to me in the discussion   00:40:51.200 --> 00:40:58.240 about pharmaceuticals was driving home the  emphasis that there is a holistic focus on   00:40:58.240 --> 00:41:04.800 safety and efficacy. These go hand in hand and  decisions must be made while considering both   00:41:04.800 --> 00:41:11.360 pieces of the picture. I would like to see that  further emphasized in the AI evaluation space. 00:41:11.360 --> 00:41:18.080 Often, we are seeing evaluations of  risk being separated from evaluations   00:41:18.080 --> 00:41:27.440 of performance or quality or efficacy, but  these two pieces of the puzzle really are   00:41:27.440 --> 00:41:33.760 not enough for us to make informed  decisions independently. And that   00:41:33.760 --> 00:41:41.040 ties back into my desire to really  also see us measuring the impacts. 00:41:41.040 --> 00:41:47.920 So we see Phase 3 trials as something that occurs  in the medical devices and pharmaceuticals field.   00:41:47.920 --> 00:41:52.640 That's not something that we are doing an  equivalent of in the AI evaluation space   00:41:52.640 --> 00:41:59.440 at this time. These are really cost intensive.  They can last years and really involve careful   00:41:59.440 --> 00:42:05.840 monitoring of that holistic picture of  safety and efficacy. And realistically,   00:42:05.840 --> 00:42:10.560 we are not going to be able to put that  on the critical path to getting specific   00:42:10.560 --> 00:42:16.400 individual AI models or AI systems vetted  before they go out into the world. However,   00:42:16.400 --> 00:42:24.800 I would love to see a world in which this sort  of work is prioritized and funded or required.   00:42:24.800 --> 00:42:31.360 Think of how, with social media, it took quite  a long time for us to understand that there are   00:42:31.360 --> 00:42:38.720 some long-term negative impacts on mental health,  and we have the opportunity now, while the AI wave   00:42:38.720 --> 00:42:44.880 is still building, to start prioritizing and  funding this sort of work. Let it run in the   00:42:44.880 --> 00:42:52.960 background and as soon as possible develop a good  understanding of the subtle, long-term effects. 00:42:52.960 --> 00:42:59.040 More broadly, I would love to see us focus on  reliability and validity of the evaluations we're   00:42:59.040 --> 00:43:06.240 conducting because trust in these decisions  and claims is important. If we don't focus   00:43:06.240 --> 00:43:11.760 on building reliable, valid, and trustworthy  evaluations, we're just going to continue to be   00:43:11.760 --> 00:43:18.320 flooded by a bunch of competing, conflicting,  and largely meaningless AI evaluations. 00:43:18.320 --> 00:43:22.480 SULLIVAN: In a number of the discussions  we've had on this podcast, we talked about   00:43:22.480 --> 00:43:28.000 how it's not just one entity that really needs  to ensure safety across the board, and I’d just   00:43:28.000 --> 00:43:33.040 love to hear from you how you think about some  of those ecosystem collaborations, and you know,   00:43:33.040 --> 00:43:38.800 from across … where we think about ourselves  as more of a platform company or places that   00:43:38.800 --> 00:43:43.760 these AI models are being deployed more at the  application level. Tell me a little bit about how   00:43:43.760 --> 00:43:49.400 you think about, sort of, stakeholders in that mix  and where responsibility lies across the board. 00:43:49.400 --> 00:43:54.560 ATALLA: It's interesting. In this age  of general-purpose AI technologies,   00:43:54.560 --> 00:44:00.080 we're often seeing one company or  organization being responsible for   00:44:00.080 --> 00:44:05.440 building the foundational model. And then  many, many other people will take that   00:44:05.440 --> 00:44:11.120 model and build it into specific products that  are designed for specific tasks and contexts. 00:44:11.120 --> 00:44:17.440 Of course, in that, we already see that  there is a responsibility of the owners   00:44:17.440 --> 00:44:22.560 of that foundational model to do some  testing of the central model before   00:44:22.560 --> 00:44:27.120 they distribute it broadly. And then  again, there is responsibility of all   00:44:27.120 --> 00:44:34.320 of the downstream individuals digesting that  and turning it into products to consider the   00:44:34.320 --> 00:44:39.520 specific contexts that they are deploying  into and how that may affect the risks we're   00:44:39.520 --> 00:44:47.040 concerned with or the types of quality and  safety and performance we need to evaluate. 00:44:47.040 --> 00:44:52.640 Again, because that field of risks  we may be concerned with is so broad,   00:44:52.640 --> 00:44:59.680 some of them also require an immense amount of  expertise. Let's think about whether AI systems   00:44:59.680 --> 00:45:06.640 can enable people to create dangerous chemicals  or dangerous weapons at home. It's not that every   00:45:06.640 --> 00:45:12.640 AI practitioner is going to have the knowledge  to evaluate this, so in some of those cases,   00:45:12.640 --> 00:45:19.600 we really need third-party experts, people  who are experts in chemistry, biology,   00:45:19.600 --> 00:45:26.800 etc., to come in and evaluate certain systems  and models for those specific risks, as well. 00:45:26.800 --> 00:45:31.680 So I think there are many reasons why  multiple stakeholders need to be involved,   00:45:31.680 --> 00:45:36.400 partly from who owns what and is responsible  for what and partly from the perspective of   00:45:36.400 --> 00:45:42.200 who has the expertise to meaningfully  construct the evaluations that we need. 00:45:42.200 --> 00:45:47.360 SULLIVAN: Well, Chad, this has just been great  to connect, and in a few of our discussions,   00:45:47.360 --> 00:45:53.040 we've done a bit of a lightning round, so I'd  love to just hear your 30-second responses   00:45:53.040 --> 00:45:58.400 to a few of these questions. Perhaps favorite  evaluation you've run so far this year? 00:45:58.400 --> 00:46:04.560 ATALLA: So I've been involved in trying to  evaluate some language models for whether   00:46:04.560 --> 00:46:10.720 they infer sensitive attributes about people.  So perhaps you're chatting with a chatbot,   00:46:10.720 --> 00:46:17.440 and it infers your religion or sexuality based  on things you're saying or how you sound, right.   00:46:17.440 --> 00:46:23.120 And in working to evaluate this, we encounter a  lot of interesting questions. Or, like, what is   00:46:23.120 --> 00:46:28.560 a sensitive attribute? What makes these attributes  sensitive, and what are the differences that make   00:46:28.560 --> 00:46:33.840 it inappropriate for an AI system to infer these  things about a person? Whereas realistically,   00:46:33.840 --> 00:46:39.920 whenever I meet a person on the street, my brain  is immediately forming first impressions and some   00:46:39.920 --> 00:46:45.600 assumptions about these people. So it's a very  interesting and thought-provoking evaluation to   00:46:46.240 --> 00:46:51.840 conduct and think about the norms that  we place upon people interacting with   00:46:51.840 --> 00:46:56.120 other people and the norms we place upon  AI systems interacting with other people. 00:46:56.120 --> 00:47:02.144 SULLIVAN: That’s fascinating! I'd love to hear  the AI buzzword you'd retire tomorrow. [LAUGHTER] 00:47:02.144 --> 00:47:08.320 ATALLA: I would love to see the term  “bias” being used less when referring   00:47:08.320 --> 00:47:15.440 to fairness-related issues and systems.  Bias happens to be a highly overloaded   00:47:15.440 --> 00:47:20.960 term in statistics and machine learning  and has a lot of technical meanings and   00:47:20.960 --> 00:47:26.800 just fails to perfectly capture  what we mean in the AI risk sense. 00:47:26.800 --> 00:47:30.280 SULLIVAN: And last one. One  metric we're not tracking enough. 00:47:30.280 --> 00:47:39.440 ATALLA: I would say over-blocking, and this comes  into that connection between the holistic picture   00:47:39.440 --> 00:47:47.440 of safety and efficacy. It's too easy to produce  systems that throw safety to the wind and focus   00:47:47.440 --> 00:47:52.560 purely on utility or achieving some goal, but  simultaneously, the other side of the picture   00:47:52.560 --> 00:47:59.760 is possible, where we can clamp down too hard  and reduce the utility of our systems and block   00:47:59.760 --> 00:48:07.520 even benign and useful outputs just because they  border on something sensitive. So it's important   00:48:07.520 --> 00:48:13.680 for us to track that over-blocking and actively  track that tradeoff between safety and efficacy. 00:48:13.680 --> 00:48:18.480 SULLIVAN: Yeah, we talk a lot about this on the  podcast, too, of how do you both make things safe   00:48:18.480 --> 00:48:24.320 but also ensure innovation can thrive, and I think  you hit the nail on the head with that last piece. 00:48:24.320 --> 00:48:25.158 [MUSIC] 00:48:25.158 --> 00:48:30.290 Well, Chad, this was really terrific. Thanks  for joining us and thanks for your work and your   00:48:30.290 --> 00:48:33.656 perspectives. And another big thanks to Daniel and  Timo for setting the stage earlier in the podcast. 00:48:33.656 --> 00:48:38.755 And to our listeners, thanks for tuning  in. You can find resources related to   00:48:38.755 --> 00:48:41.440 this podcast in the show notes. And  if you want to learn more about how   00:48:41.440 --> 00:48:47.360 Microsoft approaches AI governance,  you can visit microsoft.com/RAI.   00:48:47.360 --> 00:49:03.440 See you next time!  00:49:03.440 --> 00:49:04.334 [MUSIC FADES]