1 00:00:00,240 --> 00:00:02,560 This is Naomi Diaz with the Becker's Digital 2 00:00:02,560 --> 00:00:05,519 Health podcast. Here's your daily industry news briefing 3 00:00:05,519 --> 00:00:07,139 for today, August 6. 4 00:00:07,679 --> 00:00:08,179 First, 5 00:00:08,639 --> 00:00:11,119 researchers from the Icahn School of Medicine at 6 00:00:11,119 --> 00:00:14,179 Mount Sinai found that widely used AI chatbots 7 00:00:14,400 --> 00:00:15,859 designed for medical support 8 00:00:16,265 --> 00:00:19,464 are vulnerable to repeating and elaborating on false 9 00:00:19,464 --> 00:00:20,364 health information. 10 00:00:21,545 --> 00:00:24,445 Published August 2 in communications medicine, 11 00:00:24,824 --> 00:00:27,704 the study tested how large language models respond 12 00:00:27,704 --> 00:00:30,664 to fictional patient scenarios containing made up medical 13 00:00:30,664 --> 00:00:31,164 terms. 14 00:00:31,619 --> 00:00:34,439 The chat bots often accepted the false information 15 00:00:34,579 --> 00:00:35,559 without question, 16 00:00:35,939 --> 00:00:38,439 generating detailed responses about nonexistent 17 00:00:38,899 --> 00:00:41,079 conditions, symptoms, or treatments. 18 00:00:42,100 --> 00:00:44,679 Researchers then introduce a simple safeguard, 19 00:00:45,255 --> 00:00:47,015 a one sentence warning at the start of 20 00:00:47,015 --> 00:00:49,434 the prompt reminding the AI that the information 21 00:00:49,575 --> 00:00:51,195 provided may be inaccurate. 22 00:00:52,134 --> 00:00:54,295 With this added prompt, the rate of errors 23 00:00:54,295 --> 00:00:55,274 drops significantly, 24 00:00:55,734 --> 00:00:57,674 suggesting that even minimal guardrails 25 00:00:58,119 --> 00:00:59,739 can help reduce misinformation 26 00:01:00,439 --> 00:01:01,979 generated by AI systems. 27 00:01:02,840 --> 00:01:04,920 The findings highlight the risk of relying on 28 00:01:04,920 --> 00:01:07,560 AI chat box for medical decision making and 29 00:01:07,560 --> 00:01:09,979 underscore the importance of prompt engineering, 30 00:01:10,604 --> 00:01:12,545 system design, and human oversight. 31 00:01:13,405 --> 00:01:15,805 According to the research team, the results support 32 00:01:15,805 --> 00:01:18,144 the need to test AI tools using fabricated 33 00:01:18,204 --> 00:01:20,784 inputs before they are deployed in clinical settings. 34 00:01:21,484 --> 00:01:23,724 Researchers plan to apply the method to de 35 00:01:23,724 --> 00:01:27,359 identified patient data and explore additional safety props 36 00:01:27,359 --> 00:01:29,620 and retrieval tools as AI technology 37 00:01:30,000 --> 00:01:31,140 continues to evolve. 38 00:01:32,560 --> 00:01:33,060 Second, 39 00:01:33,439 --> 00:01:36,420 the venture capital arms of UPMC and UChicago 40 00:01:36,640 --> 00:01:39,165 Medicine have invested in a health care AI 41 00:01:39,245 --> 00:01:41,805 startup as a part of a $55,000,000 42 00:01:41,805 --> 00:01:43,344 series c funding round. 43 00:01:44,045 --> 00:01:44,545 UPMC 44 00:01:45,245 --> 00:01:49,025 enterprises and UCM ventures back all Ultramix, 45 00:01:49,564 --> 00:01:52,284 which says it offers its first FDA cleared 46 00:01:52,284 --> 00:01:53,185 Medicare reimbursed 47 00:01:53,645 --> 00:01:54,305 AI technology 48 00:01:54,799 --> 00:01:57,060 to aid in detecting cardiac amyloidosis 49 00:01:57,439 --> 00:02:00,180 and heart failure with preserved ejection fraction. 50 00:02:00,880 --> 00:02:03,520 The company developed the tool in collaboration with 51 00:02:03,520 --> 00:02:04,420 Mayo Clinic. 52 00:02:05,680 --> 00:02:09,040 UltraMix other health system partners include university hospitals 53 00:02:09,040 --> 00:02:09,699 and Northwestern 54 00:02:10,080 --> 00:02:10,580 Medicine. 55 00:02:11,015 --> 00:02:13,735 The startup says this platform helps hospitals cut 56 00:02:13,735 --> 00:02:14,235 unnecessary 57 00:02:14,534 --> 00:02:17,675 tests, enhance workflows, and start treatments earlier. 58 00:02:18,855 --> 00:02:19,355 Third, 59 00:02:19,974 --> 00:02:22,715 health systems are increasing their adoption of Microsoft's 60 00:02:22,935 --> 00:02:25,115 Ambient AI for clinical documentation. 61 00:02:26,159 --> 00:02:27,520 Here are four things to know from a 62 00:02:27,520 --> 00:02:29,780 July 30 Microsoft earnings call. 63 00:02:30,159 --> 00:02:33,360 First, Microsoft had a breakout physical 2025 64 00:02:33,360 --> 00:02:35,199 for the use of its Dragon co pilot, 65 00:02:35,199 --> 00:02:38,560 AKA DAX, which ambiently listens to patient visits 66 00:02:38,560 --> 00:02:41,060 and drafts clinical notes for the EHR, said 67 00:02:41,194 --> 00:02:43,375 Microsoft chair and CEO, Sato Nadala. 68 00:02:44,314 --> 00:02:44,814 Second, 69 00:02:45,355 --> 00:02:47,594 health care customers employ the tool to document 70 00:02:47,594 --> 00:02:48,635 over 13,000,000 71 00:02:48,635 --> 00:02:50,955 patient encounters in the most recent quarter, up 72 00:02:50,955 --> 00:02:53,055 nearly seven times year over year. 73 00:02:53,835 --> 00:02:54,335 Third, 74 00:02:54,819 --> 00:02:57,219 at Mercy, more than 1,000 physicians are now 75 00:02:57,219 --> 00:02:58,760 using Dragon Co pilot. 76 00:02:59,620 --> 00:03:02,980 Four, mister Nadella said of Mercy, quote, they 77 00:03:02,980 --> 00:03:05,300 have saved more than a 100,000 78 00:03:05,300 --> 00:03:07,300 to date and plan to expand to all 79 00:03:07,300 --> 00:03:08,520 5,000 providers. 80 00:03:08,955 --> 00:03:11,034 As one physician put it, the best thing 81 00:03:11,034 --> 00:03:13,275 to happen to my practice in ten years, 82 00:03:13,275 --> 00:03:14,014 end quote. 83 00:03:14,634 --> 00:03:16,235 If you would like the latest health IT 84 00:03:16,235 --> 00:03:18,394 and digital health industry news delivered to your 85 00:03:18,394 --> 00:03:19,795 inbox every morning, 86 00:03:20,195 --> 00:03:22,675 subscribe to the Becker CIO and health IT 87 00:03:22,675 --> 00:03:27,975 review e newsletter through our website at ww.beckershospitalreview.com.