WEBVTT

1
00:00:00.160 --> 00:00:02.399
- Everyone. This is Molly Gammel, Becker's Healthcare. Thank

2
00:00:02.399 --> 00:00:03.679
- you so much for tuning in to the

3
00:00:03.679 --> 00:00:06.160
- Becker's Healthcare podcast series. Today, we're going to

4
00:00:06.160 --> 00:00:07.700
- talk about agentic orchestration,

5
00:00:08.000 --> 00:00:09.939
- a new approach to enterprise AI.

6
00:00:10.240 --> 00:00:12.820
- Joining me for today's discussion is Munjal Shah,

7
00:00:12.964 --> 00:00:16.005
- cofounder and CEO of Hippocratic AI. Munjal, thanks

8
00:00:16.005 --> 00:00:17.065
- for joining me today.

9
00:00:17.445 --> 00:00:20.744
- Thanks for having me, Molly. So agentic orchestration,

10
00:00:20.885 --> 00:00:22.725
- Munjal, this is a a new term to

11
00:00:22.725 --> 00:00:24.585
- me, but it's something you and the Hippocratic

12
00:00:24.804 --> 00:00:26.404
- AI team have a a lot of depth

13
00:00:26.404 --> 00:00:27.545
- in, and you're also

14
00:00:28.160 --> 00:00:28.660
- launching

15
00:00:29.039 --> 00:00:32.079
- or or agentic orchestrators or this orchestration layer

16
00:00:32.079 --> 00:00:34.179
- that moves from individual AI agents

17
00:00:34.560 --> 00:00:38.000
- who handle discrete tasks to coordinated teams of

18
00:00:38.000 --> 00:00:40.500
- agents managing more complex workflows.

19
00:00:41.135 --> 00:00:43.774
- What does that evolution look like in practice,

20
00:00:43.774 --> 00:00:45.695
- and and what challenges are you seeing in

21
00:00:45.695 --> 00:00:47.534
- health care that really require a shift like

22
00:00:47.534 --> 00:00:48.034
- this?

23
00:00:48.494 --> 00:00:50.094
- Yeah. You know, I think when we started

24
00:00:50.094 --> 00:00:52.174
- the company, we thought we were gonna build

25
00:00:52.174 --> 00:00:52.835
- an LLM.

26
00:00:53.280 --> 00:00:55.439
- And then we all realized that nobody wants

27
00:00:55.439 --> 00:00:57.460
- to take an LLM and and have to

28
00:00:57.759 --> 00:01:00.000
- turn it into something you use for your

29
00:01:00.000 --> 00:01:00.739
- health care,

30
00:01:01.439 --> 00:01:03.679
- tasks. And so we actually were one of

31
00:01:03.679 --> 00:01:05.200
- the first to say we're gonna build an

32
00:01:05.200 --> 00:01:05.700
- agent.

33
00:01:06.204 --> 00:01:07.484
- And to do that, we had to fine

34
00:01:07.484 --> 00:01:09.805
- tune an LOM to be able to provide

35
00:01:09.805 --> 00:01:12.204
- the underlying the underpinnings. We had to

36
00:01:12.924 --> 00:01:15.484
- we then announced a set of agentic skills.

37
00:01:15.484 --> 00:01:16.844
- We said, oh, you know, you need a

38
00:01:16.844 --> 00:01:18.600
- skill to be able to talk to patients

39
00:01:18.600 --> 00:01:20.600
- that are post stroke and and have slurred

40
00:01:20.600 --> 00:01:21.959
- speech, and you need a skill to be

41
00:01:21.959 --> 00:01:24.280
- able to deal with patients that have mild

42
00:01:24.280 --> 00:01:26.380
- cognitive impairment and need a lot of reminding.

43
00:01:26.840 --> 00:01:28.439
- And so we then announced that we were

44
00:01:28.439 --> 00:01:30.120
- building a bunch of skills, and we built

45
00:01:30.120 --> 00:01:31.420
- hundreds of them, frankly,

46
00:01:31.880 --> 00:01:33.799
- to deal with things we saw in real

47
00:01:33.799 --> 00:01:35.694
- life, you know, in the I think we

48
00:01:35.694 --> 00:01:37.314
- now have 250

49
00:01:37.775 --> 00:01:40.174
- some odd million patient interactions, and and we

50
00:01:40.174 --> 00:01:41.694
- just saw a lot of problems that had

51
00:01:41.694 --> 00:01:42.435
- to be solved.

52
00:01:42.974 --> 00:01:44.974
- But as we did that, we realized that

53
00:01:44.974 --> 00:01:46.114
- isn't quite

54
00:01:46.950 --> 00:01:49.430
- how health care like, it's not enough to

55
00:01:49.430 --> 00:01:51.430
- just give you an agent to do x

56
00:01:51.430 --> 00:01:53.430
- because then you gotta say, okay. Run this

57
00:01:53.430 --> 00:01:55.670
- agent now. Oh, wait. I want another agent.

58
00:01:55.670 --> 00:01:58.170
- So let's take something like readmission prevention.

59
00:01:58.915 --> 00:02:00.454
- You know, it isn't just

60
00:02:01.314 --> 00:02:03.974
- one agent that does a discharge call.

61
00:02:04.594 --> 00:02:07.734
- It might be an agent inpatient that is

62
00:02:07.795 --> 00:02:09.655
- educating you before you leave.

63
00:02:10.354 --> 00:02:12.194
- It might be one you know, we're actually

64
00:02:12.194 --> 00:02:14.620
- working on putting our our agents into a

65
00:02:14.620 --> 00:02:17.259
- robot that might bring meds to beds for

66
00:02:17.259 --> 00:02:17.759
- you

67
00:02:18.139 --> 00:02:20.620
- and actually deliver the medication to you, the

68
00:02:20.620 --> 00:02:22.459
- first dose or the second dose, which is

69
00:02:22.459 --> 00:02:25.739
- common. Right? Because that helps to ensure patients

70
00:02:25.739 --> 00:02:27.199
- will start taking their medication.

71
00:02:28.064 --> 00:02:30.064
- We then will build another agent that'll call

72
00:02:30.064 --> 00:02:32.564
- you, you know, at some regular interval

73
00:02:33.185 --> 00:02:35.264
- that your care plan says. And so it

74
00:02:35.264 --> 00:02:36.625
- says, you know, on day three, you're supposed

75
00:02:36.625 --> 00:02:38.144
- to change your band, and it's really important

76
00:02:38.144 --> 00:02:39.504
- that you're you know, so you don't get

77
00:02:39.504 --> 00:02:40.004
- infected.

78
00:02:40.480 --> 00:02:42.240
- Like, our AI will make sure we call.

79
00:02:42.240 --> 00:02:44.560
- Now what is all that? That is a

80
00:02:44.560 --> 00:02:47.760
- group of agents designed and focused on creating

81
00:02:47.760 --> 00:02:48.420
- an outcome.

82
00:02:49.200 --> 00:02:50.960
- And that is what we really call an

83
00:02:50.960 --> 00:02:52.819
- orchestrator now, which is that

84
00:02:53.245 --> 00:02:55.325
- we think that, you know, we're not here

85
00:02:55.325 --> 00:02:58.685
- just to deliver agents that do tasks that

86
00:02:58.685 --> 00:03:00.625
- may or may not deliver the outcome.

87
00:03:01.165 --> 00:03:03.885
- You know, health care wants outcomes. We want

88
00:03:03.885 --> 00:03:05.425
- to change patient lives,

89
00:03:05.830 --> 00:03:08.229
- and we want to improve KPIs on which

90
00:03:08.229 --> 00:03:10.469
- we measure how well we're delivering the quality

91
00:03:10.469 --> 00:03:11.129
- of care.

92
00:03:11.590 --> 00:03:14.310
- And it's time to align this AI world

93
00:03:14.310 --> 00:03:15.050
- with that.

94
00:03:15.830 --> 00:03:17.909
- So many health systems have already seen success

95
00:03:17.909 --> 00:03:20.534
- using AI for individual workflows, but orchestration,

96
00:03:24.534 --> 00:03:27.415
- patient journeys. If I'm in a CEO or

97
00:03:27.415 --> 00:03:29.915
- COO shoes and looking through their perspective,

98
00:03:30.615 --> 00:03:33.175
- Munjal, what are some new operational opportunities that

99
00:03:33.175 --> 00:03:35.675
- creates that weren't possible with stand alone agents?

100
00:03:36.520 --> 00:03:38.280
- You know, I think, you know, COO or

101
00:03:38.280 --> 00:03:38.780
- CEO,

102
00:03:39.159 --> 00:03:41.479
- you love to manage by objective. Right? Please

103
00:03:41.479 --> 00:03:42.860
- lower the readmission rate.

104
00:03:43.240 --> 00:03:44.780
- Please get my loss to follow-up

105
00:03:45.400 --> 00:03:47.960
- closed and and help us increase better care

106
00:03:47.960 --> 00:03:48.840
- for these patients.

107
00:03:49.240 --> 00:03:50.460
- Please avoid leakage.

108
00:03:50.794 --> 00:03:52.495
- And I think that for the first time,

109
00:03:52.555 --> 00:03:53.055
- orchestrators

110
00:03:53.514 --> 00:03:56.715
- offer a solution based approach to using AgenTic

111
00:03:56.715 --> 00:03:57.215
- AI.

112
00:03:57.514 --> 00:04:00.634
- You just indicate the objective you want and

113
00:04:00.634 --> 00:04:02.715
- the solution you're really after, the area of

114
00:04:02.715 --> 00:04:04.939
- focus, and and you do that. So we're

115
00:04:04.939 --> 00:04:07.659
- now providing orchestrators that help you improve all

116
00:04:07.659 --> 00:04:09.900
- of your onboarding, improve all of your leakage,

117
00:04:09.900 --> 00:04:11.680
- improve all of your loss to follow-up,

118
00:04:12.060 --> 00:04:13.120
- improve your pharmacy,

119
00:04:13.819 --> 00:04:17.439
- business, improve your inpatient experience, your ambulatory experience,

120
00:04:18.584 --> 00:04:21.305
- reduce your readmissions, and improve your heated scores

121
00:04:21.305 --> 00:04:23.225
- and your stars. And so, you know, if

122
00:04:23.225 --> 00:04:24.824
- you think about it, it allows you to

123
00:04:24.824 --> 00:04:27.464
- operate on an objective level rather than just

124
00:04:27.464 --> 00:04:29.944
- on a task level. And I think until

125
00:04:29.944 --> 00:04:31.165
- now, we really couldn't

126
00:04:31.704 --> 00:04:32.204
- really

127
00:04:32.550 --> 00:04:34.629
- ask our AI to achieve an objective and

128
00:04:34.629 --> 00:04:36.550
- know it would go and figure out all

129
00:04:36.550 --> 00:04:38.790
- the things needed to to do that. There's

130
00:04:38.790 --> 00:04:41.050
- actually three systems where they had us do

131
00:04:41.189 --> 00:04:44.089
- discharge calls and try to avoid, readmissions.

132
00:04:44.854 --> 00:04:47.334
- And we realized how many different agents we

133
00:04:47.334 --> 00:04:49.415
- had to build to be able to achieve

134
00:04:49.415 --> 00:04:50.074
- the goal.

135
00:04:50.454 --> 00:04:52.375
- And we were in in actually, two of

136
00:04:52.375 --> 00:04:54.055
- the systems, both of them, we ended up

137
00:04:54.055 --> 00:04:57.334
- lowering readmission rates by 4%. One was 18

138
00:04:57.334 --> 00:04:57.914
- to 14,

139
00:04:58.610 --> 00:05:00.290
- which if you know anything about readmission rate

140
00:05:00.290 --> 00:05:03.089
- reduction was so big a change, nobody believed

141
00:05:03.089 --> 00:05:03.589
- it.

142
00:05:04.129 --> 00:05:06.449
- Like, we didn't believe it. Our client health

143
00:05:06.449 --> 00:05:09.329
- system client didn't believe it. And in fact,

144
00:05:09.329 --> 00:05:11.669
- they made us rerun the number seven times

145
00:05:11.730 --> 00:05:12.709
- on our side.

146
00:05:13.145 --> 00:05:15.465
- We can we actually even we reran them

147
00:05:15.465 --> 00:05:16.824
- because we're like, this can't be true. This

148
00:05:16.824 --> 00:05:17.884
- has gotta be impossible.

149
00:05:18.504 --> 00:05:20.845
- And it turned out that I mean, after

150
00:05:20.985 --> 00:05:22.745
- seven times, we all realized, wow. It it

151
00:05:22.745 --> 00:05:24.025
- was really true. And then we did it

152
00:05:24.025 --> 00:05:25.785
- again in another health system and saw another

153
00:05:25.785 --> 00:05:27.930
- four point drop. And that's when we realized,

154
00:05:27.930 --> 00:05:30.430
- oh, wow. Like, this is actually working.

155
00:05:31.050 --> 00:05:33.370
- Absolutely. A a range of of settings and

156
00:05:33.370 --> 00:05:35.689
- needs that you just highlighted there. And I'm

157
00:05:35.689 --> 00:05:37.930
- I'm putting myself in the in the position

158
00:05:37.930 --> 00:05:39.610
- of our our audience who I think hearing

159
00:05:39.610 --> 00:05:41.324
- some of these things, Minjal, they might share

160
00:05:41.324 --> 00:05:43.404
- that early sense of of disbelief or just

161
00:05:43.404 --> 00:05:46.125
- like they're we need to reorient ourselves thinking

162
00:05:46.125 --> 00:05:48.365
- about this in terms of what orchestration can

163
00:05:48.365 --> 00:05:48.865
- do.

164
00:05:49.564 --> 00:05:51.485
- You've shared an example. You talked about some

165
00:05:51.485 --> 00:05:53.790
- of the discharge calls, but another type of

166
00:05:53.790 --> 00:05:55.710
- call I've heard about is this example of

167
00:05:55.710 --> 00:05:58.750
- an academic medical center where your agents contacted

168
00:05:58.750 --> 00:05:59.949
- more than 1,700

169
00:05:59.949 --> 00:06:02.930
- patients who had open CT scan orders in

170
00:06:03.069 --> 00:06:05.310
- three days, and the result was two hundred

171
00:06:05.310 --> 00:06:07.730
- and forty nine scheduled lung cancer screenings.

172
00:06:09.014 --> 00:06:10.455
- So that was that was one agent in

173
00:06:10.455 --> 00:06:11.194
- one workflow.

174
00:06:11.574 --> 00:06:13.595
- What becomes possible when the

175
00:06:13.975 --> 00:06:16.694
- suites of agents are working together across an

176
00:06:16.694 --> 00:06:17.675
- entire enterprise?

177
00:06:18.215 --> 00:06:19.975
- Yeah. Let's let's take that as, like we

178
00:06:19.975 --> 00:06:22.314
- call that the kinda lost to follow-up orchestrator.

179
00:06:22.930 --> 00:06:24.289
- You know, that was one list. By the

180
00:06:24.289 --> 00:06:25.990
- way, of those 250

181
00:06:26.289 --> 00:06:29.089
- folks who did this, scheduled it, a 180

182
00:06:29.089 --> 00:06:31.990
- showed up. We found nine significant findings.

183
00:06:33.169 --> 00:06:35.649
- In fact, there's probably gonna be three cancers

184
00:06:35.649 --> 00:06:36.709
- found out of that.

185
00:06:37.475 --> 00:06:38.995
- Oh, I know there was one where they

186
00:06:38.995 --> 00:06:40.514
- just had to they caught it so early,

187
00:06:40.514 --> 00:06:42.214
- they were able to just do a resection

188
00:06:42.675 --> 00:06:45.714
- and didn't need to actually do, more,

189
00:06:46.115 --> 00:06:46.615
- invasive,

190
00:06:47.555 --> 00:06:48.055
- and,

191
00:06:49.314 --> 00:06:50.915
- you know, chemo and other things. So it

192
00:06:50.915 --> 00:06:52.650
- was great, that we caught it so early.

193
00:06:52.650 --> 00:06:54.009
- So, I mean, you know, we're this is

194
00:06:54.009 --> 00:06:56.189
- delivering real good. But

195
00:06:56.569 --> 00:06:58.650
- our goal isn't just to do it for

196
00:06:58.650 --> 00:06:59.150
- CT

197
00:06:59.689 --> 00:07:01.529
- lung cancer screenings. We should do it for

198
00:07:01.529 --> 00:07:02.029
- everything.

199
00:07:02.730 --> 00:07:04.590
- Right? Lost a follow-up on colonoscopies.

200
00:07:05.210 --> 00:07:06.590
- Lost a follow-up on

201
00:07:06.955 --> 00:07:08.795
- mammograms. You know, some of those are care

202
00:07:08.795 --> 00:07:10.495
- gaps, but also lost a follow-up

203
00:07:10.955 --> 00:07:13.194
- on many there's many things that patients don't

204
00:07:13.194 --> 00:07:15.595
- get done. And now some of that isn't

205
00:07:15.595 --> 00:07:16.574
- just about

206
00:07:17.355 --> 00:07:19.935
- calling the patient and persuading them and convincing

207
00:07:19.995 --> 00:07:21.889
- him to do it. You know, that that

208
00:07:21.889 --> 00:07:23.649
- would be fine. We have an orchestrator above

209
00:07:23.649 --> 00:07:25.329
- of, like, five. But now think about this.

210
00:07:25.329 --> 00:07:26.850
- What if the patient has two things that

211
00:07:26.850 --> 00:07:28.689
- they have to do? You don't wanna call

212
00:07:28.689 --> 00:07:30.689
- them twice with two different agents. You wanna

213
00:07:30.689 --> 00:07:32.550
- avoid the abrasion and call them once.

214
00:07:33.250 --> 00:07:34.290
- Second, what if,

215
00:07:34.769 --> 00:07:35.990
- the order is expired?

216
00:07:37.704 --> 00:07:39.464
- Now you wanna follow-up with the doctor and

217
00:07:39.464 --> 00:07:40.904
- say, you know, it expired, but I don't

218
00:07:40.904 --> 00:07:42.664
- think it expired because you meant for it

219
00:07:42.664 --> 00:07:44.345
- to expire. It just expired because the guy

220
00:07:44.345 --> 00:07:45.785
- didn't get it done in so long. Do

221
00:07:45.785 --> 00:07:47.865
- you really think that mister Shaw still needs

222
00:07:47.865 --> 00:07:48.845
- to get that done?

223
00:07:49.610 --> 00:07:52.490
- And we could then activate and get a,

224
00:07:53.210 --> 00:07:56.029
- additional kind of just refresh of the order.

225
00:07:56.490 --> 00:07:58.089
- And then on top of that, there's another

226
00:07:58.089 --> 00:08:00.410
- agent that we will build that'll keep track

227
00:08:00.410 --> 00:08:01.629
- of all of your capacity.

228
00:08:02.224 --> 00:08:04.305
- Okay. We're not gonna call the lung cancer

229
00:08:04.305 --> 00:08:06.384
- folks anymore this week. We're gonna wait a

230
00:08:06.384 --> 00:08:07.204
- couple weeks.

231
00:08:08.144 --> 00:08:09.764
- Why? Because there's no slots.

232
00:08:10.224 --> 00:08:12.544
- Mhmm. Mhmm. You're you're full. So there's no

233
00:08:12.544 --> 00:08:14.144
- point in calling. We're not gonna be able

234
00:08:14.144 --> 00:08:15.284
- to find any slots.

235
00:08:15.959 --> 00:08:18.519
- And so but, oh, but our our CT

236
00:08:18.519 --> 00:08:20.920
- machine or our other machine, our MRI is

237
00:08:20.920 --> 00:08:23.259
- available, but not our CT or vice versa,

238
00:08:23.800 --> 00:08:25.980
- or our bone density machine is available.

239
00:08:26.519 --> 00:08:29.535
- Right? Our DXA scan's available. So let's go

240
00:08:29.535 --> 00:08:31.375
- and follow-up with all of the women who

241
00:08:31.375 --> 00:08:33.134
- had had hip fractures who we now wanna

242
00:08:33.134 --> 00:08:34.495
- make sure they've come back in and got

243
00:08:34.495 --> 00:08:36.034
- another bone density scan.

244
00:08:36.414 --> 00:08:37.934
- In fact, we're doing this use case for

245
00:08:37.934 --> 00:08:39.554
- a customer of ours right now,

246
00:08:40.335 --> 00:08:41.134
- going after,

247
00:08:41.840 --> 00:08:43.940
- low bone density in women in particular

248
00:08:44.320 --> 00:08:45.759
- and trying to encourage them to get a

249
00:08:45.759 --> 00:08:48.879
- bone density scan. And so so imagine now

250
00:08:48.879 --> 00:08:50.179
- you have a capacity

251
00:08:50.879 --> 00:08:53.540
- or agent combined with each of the individual

252
00:08:53.600 --> 00:08:56.394
- loss to follow-up agents combined with the consolidation

253
00:08:56.695 --> 00:08:58.315
- agent that's putting it all together,

254
00:08:58.855 --> 00:09:00.554
- to avoid abrasion. And

255
00:09:01.014 --> 00:09:02.774
- and then, obviously, you have a scheduling agent

256
00:09:02.774 --> 00:09:04.855
- that'll do all of the actual scheduling of

257
00:09:04.855 --> 00:09:07.500
- the thing, plus one that follow-up with the

258
00:09:07.500 --> 00:09:09.340
- actual physician if we need to change the

259
00:09:09.340 --> 00:09:11.040
- order in some way or get it refreshed.

260
00:09:11.100 --> 00:09:11.759
- And so

261
00:09:12.779 --> 00:09:13.919
- that is an orchestrator.

262
00:09:14.379 --> 00:09:15.919
- We're trying to get to the outcome.

263
00:09:17.100 --> 00:09:18.620
- And and this is work that is much

264
00:09:18.620 --> 00:09:19.120
- more

265
00:09:20.054 --> 00:09:23.414
- complex, conditional. It's it's multipart. It's open to

266
00:09:23.414 --> 00:09:23.914
- interpretation

267
00:09:24.215 --> 00:09:25.754
- in some ways like you just highlighted.

268
00:09:26.455 --> 00:09:28.375
- So thanks for those examples. I think it's

269
00:09:28.375 --> 00:09:30.934
- really helpful to start to understand how this

270
00:09:30.934 --> 00:09:32.855
- differs from the agents that health systems might

271
00:09:32.855 --> 00:09:34.315
- be used to in current day.

272
00:09:35.049 --> 00:09:37.129
- Another element of all this work and and

273
00:09:37.129 --> 00:09:39.769
- scaling AI across an organization, it just naturally

274
00:09:39.769 --> 00:09:42.829
- raises questions about governance, about trust. Hippocratic

275
00:09:43.289 --> 00:09:46.250
- AI's Polaris uses a primary model alongside more

276
00:09:46.250 --> 00:09:49.304
- than 30 supervising models that are validated by

277
00:09:49.384 --> 00:09:51.884
- thousands of clinicians. So this is clearly something

278
00:09:52.424 --> 00:09:54.605
- you and your company take quite seriously.

279
00:09:55.464 --> 00:09:57.705
- I guess, Munjal, at a broader level, when

280
00:09:57.705 --> 00:09:59.784
- you look around in the environment health care

281
00:09:59.784 --> 00:10:02.379
- leaders are operating within today, what is a

282
00:10:02.379 --> 00:10:04.459
- a common misconception or something that we might

283
00:10:04.459 --> 00:10:07.259
- overlook about AI safety that warrants a closer

284
00:10:07.259 --> 00:10:07.759
- look?

285
00:10:09.019 --> 00:10:10.940
- So I said two things on that. One

286
00:10:10.940 --> 00:10:12.799
- is, you know, trust is is,

287
00:10:13.659 --> 00:10:14.159
- compounding.

288
00:10:14.754 --> 00:10:16.995
- You know? So we didn't ask everybody to

289
00:10:16.995 --> 00:10:19.475
- trust our orchestrators that obviously are a level

290
00:10:19.475 --> 00:10:20.134
- of abstraction

291
00:10:20.514 --> 00:10:23.814
- and independence that's that's a little bit more,

292
00:10:23.954 --> 00:10:25.254
- right, than you've seen before.

293
00:10:25.554 --> 00:10:27.414
- We first said trust our agent.

294
00:10:28.160 --> 00:10:30.080
- Now that you trust our agents, now you

295
00:10:30.080 --> 00:10:32.019
- can trust when we combine our agents.

296
00:10:32.559 --> 00:10:34.240
- So it's almost like I'm not gonna give

297
00:10:34.240 --> 00:10:36.639
- you the molecule without first you trusting every

298
00:10:36.639 --> 00:10:38.340
- atom that's going in the molecule.

299
00:10:39.600 --> 00:10:41.679
- And so if you think about, like, we

300
00:10:41.679 --> 00:10:43.195
- we and then and then we told you

301
00:10:43.195 --> 00:10:45.034
- how we built the atoms themselves. We're like,

302
00:10:45.034 --> 00:10:46.554
- look. We tested them. We did this. We

303
00:10:46.554 --> 00:10:48.235
- did that. So I I think that you

304
00:10:48.235 --> 00:10:50.014
- couldn't have gotten here till now.

305
00:10:50.714 --> 00:10:52.554
- You had to start with, alright. Here's how

306
00:10:52.554 --> 00:10:54.075
- we took the LOM. Here's how we train

307
00:10:54.075 --> 00:10:56.500
- the LOM. Here's how we then built it

308
00:10:56.500 --> 00:10:58.259
- into an agent. Here's how we validated the

309
00:10:58.259 --> 00:11:00.179
- agent. Here's how we now run a bunch

310
00:11:00.179 --> 00:11:01.799
- of you know, I think we've now done

311
00:11:02.740 --> 00:11:04.580
- we've done this 250,000,000

312
00:11:04.580 --> 00:11:07.539
- patient interactions without one safety incident. Because we

313
00:11:07.539 --> 00:11:10.019
- do escalate to humans when necessary, we've escalated

314
00:11:10.019 --> 00:11:12.164
- to humans 38,000

315
00:11:12.625 --> 00:11:13.125
- times.

316
00:11:13.504 --> 00:11:14.004
- Wow.

317
00:11:14.384 --> 00:11:17.205
- So we know what happens when it happens,

318
00:11:17.264 --> 00:11:19.664
- how to ensure, you know, and we've avoided

319
00:11:19.664 --> 00:11:21.764
- any significant safety issues. So it's like

320
00:11:22.544 --> 00:11:24.699
- that's that piece. But I think that I

321
00:11:24.699 --> 00:11:28.159
- think when people are having conceptions about safety,

322
00:11:28.459 --> 00:11:30.720
- I I think it's in the abstract mostly.

323
00:11:31.659 --> 00:11:33.600
- It's like you've never actually

324
00:11:34.220 --> 00:11:36.220
- done the thing, and you're worried about its

325
00:11:36.220 --> 00:11:37.820
- safety or not. But once you do it,

326
00:11:37.820 --> 00:11:39.904
- you realize, a, it's not it's not a

327
00:11:39.904 --> 00:11:42.004
- worry in some things you were worried about.

328
00:11:42.225 --> 00:11:44.064
- And b, it is actually so you have

329
00:11:44.064 --> 00:11:44.644
- to be,

330
00:11:45.024 --> 00:11:46.625
- you know, watchful in this other area, but

331
00:11:46.625 --> 00:11:48.144
- you didn't know it because you've never done

332
00:11:48.144 --> 00:11:50.944
- it. Mhmm. Like, think about when you before

333
00:11:50.944 --> 00:11:52.164
- you ever went skiing,

334
00:11:52.629 --> 00:11:54.230
- would you have known, like, what you gotta

335
00:11:54.230 --> 00:11:55.750
- worry about when you ski to not get

336
00:11:55.750 --> 00:11:58.230
- injured? Right? Like, you know, now whenever I

337
00:11:58.230 --> 00:11:59.909
- ski, by the way, like, I I'm always

338
00:11:59.909 --> 00:12:01.850
- off the slope by, like, 01:00,

339
00:12:02.470 --> 00:12:04.629
- 02:00 at the moment. Right? Because I always

340
00:12:04.629 --> 00:12:06.309
- feel like you get injured in the last,

341
00:12:06.309 --> 00:12:07.129
- like, hour

342
00:12:07.615 --> 00:12:09.295
- of your skis thing when the snow is

343
00:12:09.295 --> 00:12:11.535
- slushy and it's horrible, at least here in

344
00:12:11.535 --> 00:12:12.915
- California. And so

345
00:12:13.375 --> 00:12:14.735
- but I wouldn't have known that if I

346
00:12:14.735 --> 00:12:16.335
- never skied. So first, I I think you

347
00:12:16.335 --> 00:12:17.855
- need to focus on that. In terms of

348
00:12:17.855 --> 00:12:20.514
- the actual things to kinda look out for,

349
00:12:21.009 --> 00:12:23.889
- I think there's a siren song right now

350
00:12:23.889 --> 00:12:25.589
- around smaller models.

351
00:12:26.209 --> 00:12:28.370
- And the siren song says, I can give

352
00:12:28.370 --> 00:12:31.569
- you a cheaper AI because I'm using smaller

353
00:12:31.569 --> 00:12:33.429
- models. I'm calling them SDMs

354
00:12:34.375 --> 00:12:36.955
- or SLMs, small language models. Okay.

355
00:12:37.335 --> 00:12:38.394
- That's not a thing.

356
00:12:39.014 --> 00:12:40.855
- The reason this was a big innovation in

357
00:12:40.855 --> 00:12:43.434
- the world was it's called a large language

358
00:12:43.575 --> 00:12:44.075
- model.

359
00:12:44.615 --> 00:12:46.855
- Mhmm. It's the large part that made it

360
00:12:46.855 --> 00:12:47.355
- smart.

361
00:12:48.080 --> 00:12:51.139
- Actually, very directly, the size of the model

362
00:12:51.279 --> 00:12:52.820
- correlates to its intelligence.

363
00:12:54.240 --> 00:12:56.720
- Like, everybody doesn't realize this. Like, the breakthrough

364
00:12:56.720 --> 00:12:59.360
- for CHAT GPT came because we crossed 30,000,000,000

365
00:12:59.360 --> 00:13:01.615
- parameters. What's a parameter? It's the number of

366
00:13:01.615 --> 00:13:03.934
- connections between the neurons and the neural network.

367
00:13:03.934 --> 00:13:06.014
- It's literally like the brain. The more connections

368
00:13:06.014 --> 00:13:08.595
- a brain has, the smarter the brain is.

369
00:13:09.454 --> 00:13:11.294
- So I'm like, great. You wanna downgrade to

370
00:13:11.294 --> 00:13:12.194
- a mouse brain?

371
00:13:12.654 --> 00:13:14.514
- Because that's kinda what you just did,

372
00:13:14.815 --> 00:13:15.315
- literally.

373
00:13:16.019 --> 00:13:18.180
- You literally just downgraded to a mouse brain

374
00:13:18.180 --> 00:13:19.940
- when you moved to an SLM. And that

375
00:13:19.940 --> 00:13:21.000
- may work fine,

376
00:13:21.379 --> 00:13:23.139
- but you know what it doesn't do? It

377
00:13:23.139 --> 00:13:26.180
- doesn't catch all the corner cases, and medicine

378
00:13:26.180 --> 00:13:28.279
- is a giant game of corner cases.

379
00:13:29.024 --> 00:13:31.605
- Mhmm. Mhmm. And, like, you know, like, diagnosing

380
00:13:31.745 --> 00:13:33.424
- UTIs in women, if you you know, we

381
00:13:33.424 --> 00:13:35.365
- don't do diagnosis. But if you did,

382
00:13:35.985 --> 00:13:38.065
- yeah. Could a little rule system get you

383
00:13:38.065 --> 00:13:40.004
- there eighty percent of the time? Sure.

384
00:13:41.269 --> 00:13:43.190
- But it's that twenty percent of the time,

385
00:13:43.190 --> 00:13:45.370
- five percent of the time, that's when,

386
00:13:46.230 --> 00:13:48.490
- you know, when the UTI is a possible

387
00:13:49.429 --> 00:13:51.509
- infection risk to the kidney. And, I mean,

388
00:13:51.509 --> 00:13:53.290
- that that that's when it gets complicated.

389
00:13:53.670 --> 00:13:56.009
- Yeah. And so I think that,

390
00:13:56.914 --> 00:13:58.534
- anyways, there's there's a,

391
00:13:59.154 --> 00:14:00.514
- one of the things I think people don't

392
00:14:00.514 --> 00:14:03.154
- realize is safety and size of model are

393
00:14:03.154 --> 00:14:04.054
- directly related.

394
00:14:04.595 --> 00:14:05.095
- Mhmm.

395
00:14:05.794 --> 00:14:07.235
- Mhmm. Thank you for expanding on that. I

396
00:14:07.235 --> 00:14:09.299
- think also just for highlighting some of the

397
00:14:09.379 --> 00:14:11.700
- the human biases we may bring to to

398
00:14:11.700 --> 00:14:14.259
- the unknown as these are now this technology

399
00:14:14.259 --> 00:14:16.740
- is capable of doing things that six months,

400
00:14:16.740 --> 00:14:19.159
- eighteen months ago just was not in existence.

401
00:14:19.220 --> 00:14:20.980
- So I think it's really helpful to to

402
00:14:20.980 --> 00:14:22.440
- keep that in mind, Munjal.

403
00:14:22.894 --> 00:14:25.615
- As we wind down here, in addition to

404
00:14:25.615 --> 00:14:26.514
- agentic orchestration,

405
00:14:27.215 --> 00:14:29.134
- you also rolled out what I understand to

406
00:14:29.134 --> 00:14:31.774
- be a self-service agent builder. Can you just

407
00:14:31.774 --> 00:14:34.095
- explain what this is for one second and

408
00:14:34.095 --> 00:14:36.190
- also what it means for the ability of

409
00:14:36.190 --> 00:14:38.429
- health care leaders to get to market and

410
00:14:38.429 --> 00:14:39.809
- production more quickly?

411
00:14:40.669 --> 00:14:42.830
- Yeah. So the the innovation of our agent

412
00:14:42.830 --> 00:14:44.429
- builder is actually not the agent builder, but

413
00:14:44.429 --> 00:14:45.490
- it's actually the validation.

414
00:14:46.110 --> 00:14:48.769
- So we developed this unique way to validate

415
00:14:48.830 --> 00:14:51.144
- agents that, you know, I haven't seen anybody

416
00:14:51.144 --> 00:14:53.305
- else do, which is we hired 7,000

417
00:14:53.305 --> 00:14:55.245
- now 7,750

418
00:14:55.945 --> 00:14:56.445
- licensed

419
00:14:56.825 --> 00:14:57.485
- US clinicians.

420
00:14:58.425 --> 00:14:59.705
- We gave them, like, you know, what we

421
00:14:59.705 --> 00:15:01.465
- call our gig app. They actually sign on

422
00:15:01.465 --> 00:15:03.305
- on their iPhone every, you know, morning, and

423
00:15:03.305 --> 00:15:04.585
- they see if there's tasks for them to

424
00:15:04.585 --> 00:15:06.059
- do, and we pay them for task. You

425
00:15:06.059 --> 00:15:08.220
- know? And the tasks are usually act as

426
00:15:08.220 --> 00:15:10.960
- a patient, call this new agent we're building,

427
00:15:11.419 --> 00:15:13.200
- and or this new orchestrator,

428
00:15:13.660 --> 00:15:14.160
- and

429
00:15:14.700 --> 00:15:16.460
- tell us if it's making mistakes and mark

430
00:15:16.460 --> 00:15:18.315
- every error. And one out of 50 times,

431
00:15:18.315 --> 00:15:19.835
- they got a human on the other line,

432
00:15:19.835 --> 00:15:21.835
- so then we got to calibrate how good

433
00:15:21.835 --> 00:15:23.675
- the error rate was compared to the kind

434
00:15:23.675 --> 00:15:25.915
- of, human error rate. We used to build

435
00:15:25.915 --> 00:15:27.915
- all of our models that way. They would

436
00:15:27.915 --> 00:15:30.075
- all get validated to this process, and we've

437
00:15:30.075 --> 00:15:32.240
- now done 700,000

438
00:15:32.320 --> 00:15:33.759
- I see almost 800,000

439
00:15:33.759 --> 00:15:35.700
- test calls with their simulations.

440
00:15:36.320 --> 00:15:38.240
- And we wouldn't release a a model until

441
00:15:38.240 --> 00:15:40.799
- it had been validated as as safer than

442
00:15:40.799 --> 00:15:43.039
- the human equivalent. We wouldn't validate we wouldn't

443
00:15:43.039 --> 00:15:45.434
- release a use case or an agent until

444
00:15:45.434 --> 00:15:47.034
- we had done that, you know, and now

445
00:15:47.034 --> 00:15:48.574
- we won't release an an orchestrator

446
00:15:48.954 --> 00:15:50.095
- until we've done that.

447
00:15:50.554 --> 00:15:52.014
- But we had to do that.

448
00:15:52.315 --> 00:15:53.834
- If you we had a little agent builder

449
00:15:53.834 --> 00:15:55.194
- where you could build your own agent, but

450
00:15:55.194 --> 00:15:57.595
- then we would, by hand, take it through

451
00:15:57.595 --> 00:15:58.894
- this validation system.

452
00:15:59.690 --> 00:16:01.610
- But now what we do is we link

453
00:16:01.610 --> 00:16:02.269
- the two.

454
00:16:02.649 --> 00:16:04.429
- So you can build your own agent.

455
00:16:05.610 --> 00:16:08.009
- You can send it. It'll automatically send it

456
00:16:08.009 --> 00:16:08.669
- for validation.

457
00:16:10.330 --> 00:16:11.789
- It'll validate it

458
00:16:12.335 --> 00:16:13.855
- and give you back the feedback. But then

459
00:16:13.855 --> 00:16:15.455
- we said, wait a minute. We don't even

460
00:16:15.455 --> 00:16:17.154
- just have to give you back the feedback.

461
00:16:17.455 --> 00:16:20.335
- We'll run another agent to rewrite your prompts

462
00:16:20.335 --> 00:16:22.595
- and your scripts based upon the feedback.

463
00:16:23.535 --> 00:16:25.394
- We'll send it for validation again,

464
00:16:26.009 --> 00:16:27.690
- and we'll do this loop for a week

465
00:16:27.690 --> 00:16:30.090
- without you even being involved and then say,

466
00:16:30.090 --> 00:16:31.690
- hey. Here's what we think is the right

467
00:16:31.690 --> 00:16:33.769
- one. Now you try it again and tell

468
00:16:33.769 --> 00:16:35.290
- us where it is, and then we'll validate

469
00:16:35.290 --> 00:16:36.190
- it one more time.

470
00:16:36.570 --> 00:16:39.290
- Wow. And so the power is really in

471
00:16:39.290 --> 00:16:41.735
- the combination of the agent builder feeding into

472
00:16:41.735 --> 00:16:44.694
- it, but then making it a continuous improving

473
00:16:44.694 --> 00:16:46.954
- loop that actually is somewhat autonomous.

474
00:16:47.654 --> 00:16:50.134
- And that's the power of this self-service is

475
00:16:50.134 --> 00:16:50.214
- is

476
00:16:50.934 --> 00:16:52.214
- you know, we probably should have come up

477
00:16:52.214 --> 00:16:53.414
- with a better name for it because it's

478
00:16:53.414 --> 00:16:55.879
- not just self-service. It's our self validating

479
00:16:57.059 --> 00:16:59.379
- way to kind of develop agents and and

480
00:16:59.379 --> 00:17:02.339
- orchestrators around that. And so, you know, I

481
00:17:02.339 --> 00:17:04.019
- I think that's what I'm most excited about

482
00:17:04.019 --> 00:17:04.839
- is is

483
00:17:05.299 --> 00:17:08.179
- we are now exporting the ability for you

484
00:17:08.179 --> 00:17:10.214
- to make safe agents, not just you to

485
00:17:10.214 --> 00:17:11.035
- make agents.

486
00:17:12.054 --> 00:17:13.494
- I'm really glad you expand on that. I

487
00:17:13.494 --> 00:17:15.255
- thought it was more of, like, just the

488
00:17:15.255 --> 00:17:17.414
- access of building an agent was gonna be

489
00:17:17.414 --> 00:17:19.654
- the value proposition there, but it's actually the

490
00:17:19.654 --> 00:17:23.194
- validation and your homegrown process of safety assurance

491
00:17:23.720 --> 00:17:24.859
- now being accessible

492
00:17:25.160 --> 00:17:26.599
- to the end user, like, in in such

493
00:17:26.599 --> 00:17:27.259
- a way.

494
00:17:27.640 --> 00:17:28.380
- Right. Exactly.

495
00:17:28.759 --> 00:17:29.259
- Exactly.

496
00:17:30.119 --> 00:17:32.440
- Manjal, you you and Hippocratic AI just continue

497
00:17:32.440 --> 00:17:33.640
- to be I I don't even know how

498
00:17:33.640 --> 00:17:35.400
- many steps ahead. I I just wanna thank

499
00:17:35.400 --> 00:17:36.680
- you so much for your time and your

500
00:17:36.680 --> 00:17:39.134
- insights today. This was a really interesting conversation.

501
00:17:39.994 --> 00:17:42.154
- Thanks for joining me. Yeah. Thanks for having

502
00:17:42.154 --> 00:17:43.695
- me, Molly. It's always a pleasure.

503
00:17:44.075 --> 00:17:45.674
- And to our audience, thank you so much

504
00:17:45.674 --> 00:17:47.355
- for tuning in. You can tune in to

505
00:17:47.355 --> 00:17:49.434
- more podcasts from Becker's Healthcare by visiting our

506
00:17:49.434 --> 00:17:52.734
- podcast page at beckershospitalreview.com.