WEBVTT

00:00:00.000 --> 00:00:05.560
[MUSIC PLAYS]

00:00:05.560 --> 00:00:11.480
ASHLEY LLORENS: I’m Ashley Llorens with Microsoft Research. 
In this podcast series, I share conversations with  

00:00:11.480 --> 00:00:16.640
fellow researchers about the latest developments 
in AI models, the work we’re doing to understand  

00:00:16.640 --> 00:00:21.880
their capabilities and limitations, and ultimately 
how innovations like these can have the greatest  

00:00:21.880 --> 00:00:29.400
benefit for humanity. Welcome to AI Frontiers.
Today, I’ll speak with Ida Momennejad. Ida works  

00:00:29.400 --> 00:00:34.280
at Microsoft Research in New York City at 
the intersection of machine learning and  

00:00:34.280 --> 00:00:40.320
human cognition and behavior. Her current work 
focuses on building and evaluating multi-agent  

00:00:40.320 --> 00:00:46.920
AI architectures, drawing from her background in 
both computer science and cognitive neuroscience.  

00:00:46.920 --> 00:00:53.320
Over the past decade, she has focused on studying 
how humans and AI agents build and use models of  

00:00:53.320 --> 00:00:55.000
their environment.
[MUSIC FADES] 

00:00:55.000 --> 00:01:02.600
Let’s dive right in. We are undergoing a paradigm 
shift where AI models and systems are starting to  

00:01:02.600 --> 00:01:10.040
exhibit characteristics that I and, of course, 
many others have described as more general  

00:01:10.040 --> 00:01:16.120
intelligence. When I say general in this context, 
I think I mean systems with abilities like  

00:01:16.120 --> 00:01:21.920
reasoning and problem-solving that can be applied 
to many different tasks, even tasks they were not  

00:01:21.920 --> 00:01:27.640
explicitly trained to perform. Despite all of 
this, I think it’s also important to admit that  

00:01:27.640 --> 00:01:34.840
we—and by we here, I mean humanity—are not very 
good at measuring general intelligence, especially  

00:01:34.840 --> 00:01:41.360
in machines. So I’m excited to dig further into 
this topic with you today, especially given  

00:01:41.360 --> 00:01:47.440
your background and insights into both human and 
machine intelligence. And so I just want to start  

00:01:47.440 --> 00:01:54.040
here: for you, Ida, what is general intelligence?
IDA MOMENNEJAD: Thank you for asking that. We  

00:01:54.040 --> 00:01:58.640
could look at general intelligence from the 
perspective of history of cognitive science and  

00:01:58.640 --> 00:02:06.600
neuroscience. And in doing so, I’d like to mention 
its discontents, as well. There was a time where  

00:02:06.600 --> 00:02:12.600
general intelligence was introduced as the idea 
of a kind of intelligence that was separate from  

00:02:12.600 --> 00:02:18.240
what you knew or the knowledge that you had on a 
particular topic. It was this general capacity to  

00:02:18.240 --> 00:02:23.960
acquire different types of knowledge and reason 
over different things. And this was at some point  

00:02:23.960 --> 00:02:29.600
known as g, and it’s still known as g. There have 
been many different kinds of critiques of this  

00:02:29.600 --> 00:02:36.480
concept because some people said that it’s very 
much focused on the idea of logic and a particular  

00:02:36.480 --> 00:02:41.560
kind of reasoning. Some people made cultural 
critiques of it. They said it’s very Western  

00:02:41.560 --> 00:02:46.600
oriented. Others said it’s very individualistic. 
It doesn’t consider collective or interpersonal  

00:02:46.600 --> 00:02:52.400
intelligence or physical intelligence. There 
are many critiques of it. But at the core of it,  

00:02:52.400 --> 00:02:57.280
there might be something useful and helpful. 
And I think the useful part is that there  

00:02:57.280 --> 00:03:04.120
could be some general ability in humans, at 
least the way that g was intended initially,  

00:03:04.120 --> 00:03:07.880
where they can learn many different things 
and reason over many different domains,  

00:03:07.880 --> 00:03:16.120
and they can transfer ability to reason over a 
particular domain to another. And then in the AGI,  

00:03:16.120 --> 00:03:21.480
or artificial general intelligence, notion of 
it, people took this idea of many different  

00:03:21.480 --> 00:03:30.560
abilities or skills for cognitive and reasoning 
and logic problem-solving at once. There have  

00:03:30.560 --> 00:03:35.200
been different iterations of what this 
means in different times. In principle,  

00:03:35.200 --> 00:03:39.840
the concept in itself does not provide the 
criteria on its own. Different people at  

00:03:39.840 --> 00:03:43.720
different times provide different criteria 
for what would be the artificial general  

00:03:43.720 --> 00:03:48.240
intelligence notion. Some people say that they 
have achieved it. Some people say we are on the  

00:03:48.240 --> 00:03:53.440
brink of achieving it. Some people say we will 
never achieve it. However, there is this idea,  

00:03:53.440 --> 00:03:58.720
if you look at it from an evolutionary and 
neuroscience and cognitive neuroscience lens,  

00:03:58.720 --> 00:04:05.160
that in evolution, intelligence has evolved 
multiple times in a way that is adaptive to the  

00:04:05.160 --> 00:04:11.680
environment. So there were organisms that needed 
to be adaptive to the environment where they were,  

00:04:11.680 --> 00:04:17.280
that intelligence has evolved in multiple 
different species, so there’s not one solution  

00:04:17.280 --> 00:04:22.480
to it, and it depends on the ecological niche 
that that particular species needed to adapt  

00:04:22.480 --> 00:04:29.200
to and survive in. And it’s very much related to 
the idea of being adaptive of certain kinds of,  

00:04:29.200 --> 00:04:34.240
different kinds of problem-solving that 
are specific to that particular ecology.  

00:04:34.240 --> 00:04:39.360
There is also this other idea that there is 
no free lunch and the no-free-lunch theorem,  

00:04:39.360 --> 00:04:45.240
that you cannot have one particular machine 
learning solution that can solve everything.  

00:04:45.240 --> 00:04:51.840
So the idea of general artificial intelligence 
in terms of an approach that can solve everything  

00:04:51.840 --> 00:04:57.840
and there is one end-to-end training that can be 
useful to solve every possible problem that it has  

00:04:57.840 --> 00:05:03.800
never seen before seems a little bit untenable 
to me, at least at this point. What does seem  

00:05:03.800 --> 00:05:09.360
tenable to me in terms of general intelligence 
is if we understand and study, the same way that  

00:05:09.360 --> 00:05:15.880
we can do it in nature, the foundational 
components of reasoning, of intelligence,  

00:05:15.880 --> 00:05:20.280
of different particular types of intelligence, 
of different particular skills—whether it has  

00:05:20.280 --> 00:05:24.720
to do with cultural accumulation of written 
reasoning and intelligence skills, whether it  

00:05:24.720 --> 00:05:31.160
has to do with logic, whether it has to do with 
planning—and then working on the particular types  

00:05:31.160 --> 00:05:37.160
of artificial agents that are capable of putting 
these particular foundational building blocks  

00:05:37.160 --> 00:05:42.840
together in order to solve problems they’ve never 
seen before. A little bit like putting Lego pieces  

00:05:42.840 --> 00:05:49.400
together. So to wrap it up, to sum up what I just 
said, the idea of general intelligence had a more  

00:05:49.400 --> 00:05:56.080
limited meaning in cognitive science, referring to 
human ability to have multiple different types of  

00:05:56.080 --> 00:06:02.080
skills for problem-solving and reasoning. Later 
on, it was also, of course, criticized in terms  

00:06:02.080 --> 00:06:09.560
of the specificity of it and ignoring different 
kinds of intelligence. In AI, this notion has  

00:06:09.560 --> 00:06:16.640
been having many different kinds of meanings. If 
we just mean it’s a kind of a toolbox of general  

00:06:16.640 --> 00:06:21.080
kinds of intelligence for something that can 
be akin to an assistant to a human, that could  

00:06:21.080 --> 00:06:26.800
make sense. But if we go too far and use it in the 
kind of absolute notion of general intelligence,  

00:06:26.800 --> 00:06:32.840
as it has to encompass all kinds of intelligence 
possible, that might be untenable. And also  

00:06:32.840 --> 00:06:38.800
perhaps we shouldn’t think about it in terms of a 
lump of one end-to-end system that can get all of  

00:06:38.800 --> 00:06:45.640
it down. Perhaps we can think about it in terms 
of understanding the different components that we  

00:06:45.640 --> 00:06:52.120
have also seen emerge in evolution in different 
species. Some of them are robust across many  

00:06:52.120 --> 00:06:57.160
different species. Some of them are more specific 
to some species with a specific ecological niche  

00:06:57.160 --> 00:07:04.800
or specific problems to solve. But I think perhaps 
it could be more helpful to find those cognitive  

00:07:04.800 --> 00:07:10.880
and other interpersonal, cultural, different 
notions of intelligence; break them down into  

00:07:10.880 --> 00:07:17.040
their foundational building blocks; and then 
see how a particular artificial intelligence  

00:07:17.040 --> 00:07:24.840
agent can bring together different skills from 
this kind of a library of intelligence skills in  

00:07:24.840 --> 00:07:31.080
order to solve problems it’s never seen before.
LLORENS: There are two concepts that jump out  

00:07:31.080 --> 00:07:39.160
at me based on what you said. One is artificial 
general intelligence and the other is humanlike  

00:07:39.160 --> 00:07:45.640
intelligence or human-level intelligence. And 
you’ve referenced the fact that, you know,  

00:07:45.640 --> 00:07:50.160
oftentimes, we equate the two or at least 
it’s not clear sometimes how the two relate  

00:07:50.160 --> 00:07:55.960
to each other. Certainly, human intelligence 
has been an important inspiration for what  

00:07:55.960 --> 00:08:01.160
we’ve done—a lot of what we’ve done—in AI 
and, in many cases, a kind of evaluation  

00:08:01.160 --> 00:08:07.040
target in terms of how we measure progress or 
performance. But I wonder if we could just back  

00:08:07.040 --> 00:08:12.600
up a minute. Artificial general intelligence 
and humanlike, human-level intelligence—how  

00:08:12.600 --> 00:08:17.560
do these two concepts relate to you?
MOMENNEJAD: Great question. I like that you  

00:08:17.560 --> 00:08:23.360
asked to me because I think it would be different 
for different people. I’ve written about this,  

00:08:23.360 --> 00:08:30.680
in fact. I think humanlike intelligence or 
human-level intelligence would require performance  

00:08:30.680 --> 00:08:38.800
that is similar to humans, at least behaviorally, 
not just in terms of what the agent gets right,  

00:08:38.800 --> 00:08:43.360
but also in terms of the kinds of mistakes and 
biases that the agent might have. It should look  

00:08:43.360 --> 00:08:49.560
like human intelligence. For instance, humans 
show primacy bias, recency bias, variety of  

00:08:49.560 --> 00:08:56.080
biases. And this seems like it’s unhelpful in 
a lot of situations. But in some situations,  

00:08:56.080 --> 00:09:02.080
it helps to come with fast and frugal solutions 
on the go. It helps to summarize certain things  

00:09:02.080 --> 00:09:07.800
or make inferences really fast that can help 
in human intelligence. For instance, there is  

00:09:07.800 --> 00:09:14.160
analogical reasoning. That is, there are different 
types of intelligence that humans do. Now, if you  

00:09:14.160 --> 00:09:20.760
look at what are tasks that are difficult and what 
are tasks that are easier for humans and compare  

00:09:20.760 --> 00:09:27.360
that to a, for instance, let’s say just a large 
language model like GPT-4, you will see whether  

00:09:27.360 --> 00:09:32.960
they find similar things simple and similar 
things difficult or not. When they don’t find  

00:09:32.960 --> 00:09:38.200
similar things easy or difficult, I think that 
we should not say that this is humanlike per se,  

00:09:38.200 --> 00:09:44.640
unless we mean for a specific task. Perhaps 
on specific sets of tasks, an agent can be,  

00:09:44.640 --> 00:09:51.080
can have human-level or humanlike intelligent 
behavior; however, if we look overall, as long  

00:09:51.080 --> 00:09:58.920
as there are particular skills that are more or 
less difficult for one or the other, it might be  

00:09:58.920 --> 00:10:05.400
not reasonable to compare them. That being said, 
there are many things that some AI agent and even  

00:10:05.400 --> 00:10:10.480
a [programming] language would be better [than] 
humans at. Does that mean that they are generally  

00:10:10.480 --> 00:10:14.840
more intelligent? No, it doesn’t because there 
are also many things that humans are far better  

00:10:14.840 --> 00:10:22.400
than AI at. The second component of this is the 
mechanisms by which humans do the intelligent  

00:10:22.400 --> 00:10:28.960
things that we do. We are very energy efficient. 
With very little amount of energy consumption,  

00:10:28.960 --> 00:10:34.200
we can solve very complicated problems. If you 
put some of us next to each other or at least  

00:10:34.200 --> 00:10:40.120
give a pen and paper to one of us, this can be 
even a lot more effective; however, the amount  

00:10:40.120 --> 00:10:46.920
of energy consumption that it takes in order for 
any machine to solve similar problems is a lot  

00:10:46.920 --> 00:10:55.160
higher. So another difference between humanlike 
intelligence or biologically inspired intelligence  

00:10:55.160 --> 00:11:01.920
and the kind of intelligence that is in silico 
is efficiency, energy efficiency in general. And  

00:11:01.920 --> 00:11:10.560
finally, the amount of data that goes into current 
state-of-[the-art] AI versus perhaps the amount of  

00:11:10.560 --> 00:11:17.960
data that a human might need to learn new tasks or 
acquire new skills seem to be also different. So  

00:11:17.960 --> 00:11:25.840
it seems like there are a number of different 
approaches to comparing human and machine  

00:11:25.840 --> 00:11:32.600
intelligence and deriving what are the criteria 
for a machine intelligence to be more humanlike.  

00:11:32.600 --> 00:11:38.160
But other than the conceptual aspect of it, it’s 
not clear that we necessarily want something  

00:11:38.160 --> 00:11:43.920
that’s entirely humanlike. Perhaps we want in 
some tasks and in some particular use cases for  

00:11:43.920 --> 00:11:50.560
the agent to be humanlike but not in everything.
LLORENS: You mentioned some of the ways in which  

00:11:50.560 --> 00:11:57.840
human intelligence is inferior or has weaknesses. 
You mentioned some of the weaknesses of human  

00:11:57.840 --> 00:12:06.320
intelligence, like recency bias. What are some 
of the weaknesses of artificial intelligence,  

00:12:06.320 --> 00:12:13.080
especially frontier systems today? You’ve recently 
published some works that have gotten into new  

00:12:13.080 --> 00:12:18.040
paradigms for evaluation, and you’ve explored some 
of these weaknesses. And so can you tell us more  

00:12:18.560 --> 00:12:27.440
about that work and about your view on this?
MOMENNEJAD: Certainly. So inspired by a very  

00:12:27.440 --> 00:12:34.000
long-standing tradition of evaluating cognitive 
capacities—those Lego pieces that bring together  

00:12:34.000 --> 00:12:41.040
intelligence that I was mentioning in humans and 
animals—I have conducted a number of experiments,  

00:12:41.040 --> 00:12:48.440
first in humans, and built reinforcement learning 
models over the past more than a decade on the  

00:12:48.440 --> 00:12:54.880
idea of multistep reasoning and planning. 
It is in the general domain of reasoning,  

00:12:54.880 --> 00:13:01.880
planning, and decision making. And I particularly 
focused on what kind of memory representations  

00:13:01.880 --> 00:13:08.800
allow brains and reinforcement learning models 
inspired by human brain and behavior to be able  

00:13:08.800 --> 00:13:15.000
to predict the future and plan the future and 
reason over the past and the future seamlessly  

00:13:15.000 --> 00:13:22.840
using the same representations. Inspired by the 
same research that goes back in tradition to  

00:13:22.840 --> 00:13:29.160
Edward Tolman’s idea of cognitive maps and 
latent learning in the early 20th century,  

00:13:29.160 --> 00:13:36.760
culminating in his very influential 1948 paper, 
“Cognitive maps in rats and men,” I sat down with  

00:13:36.760 --> 00:13:43.040
a couple of colleagues last year—exactly this 
time, probably—and we worked on figuring out if  

00:13:43.040 --> 00:13:51.200
we can devise similar experiments to that in order 
to test cognitive maps and planning and multistep  

00:13:51.200 --> 00:13:57.120
reasoning abilities in large language models. So 
I first turned some of the experiments that I had  

00:13:57.120 --> 00:14:01.840
conducted in humans and some of the experiments 
that were done by Edward Tolman on the topic in  

00:14:01.840 --> 00:14:09.920
rodents and turned them into prompts for ChatGPT. 
That’s where I started, with GPT-4. The reason I  

00:14:09.920 --> 00:14:16.360
did that was that I wanted to make sure that I 
will create some prompts that have not been in  

00:14:16.360 --> 00:14:22.280
the training set. My experiments, although the 
papers have been published, the stimuli of the  

00:14:22.280 --> 00:14:27.440
experiments were not linguistic. They were 
visual sequences that the human would see,  

00:14:27.440 --> 00:14:32.080
and they would have to have some reinforcement 
learning and learn from the sequences to make  

00:14:32.080 --> 00:14:37.120
inferences about relationships between different 
states and find what is the path that would  

00:14:37.120 --> 00:14:44.400
give them optimal rewards. Very simple human 
reinforcement learning paradigms. However, with  

00:14:44.400 --> 00:14:51.080
different kind of structures. The inspirations 
that I had drawn from the cognitive maps works  

00:14:51.080 --> 00:14:58.400
by Edward Tolman and others was in this idea that 
in order for a creature, whether it’s a rodent,  

00:14:58.400 --> 00:15:04.000
a human, or a machine, to be able to reason in 
[multiple] steps, plan, and have cognitive maps,  

00:15:04.000 --> 00:15:09.840
which is simply a representation of the 
relational structure of the environment,  

00:15:09.840 --> 00:15:16.160
in order for a creature to have these abilities or 
these capacities, it means that the creature needs  

00:15:16.160 --> 00:15:23.480
to be sensitive and adaptive to local changes 
in the environment. So I designed the, sort of,  

00:15:23.480 --> 00:15:33.000
the initial prompts and recruited a number of very 
smart and generous-with-their-time colleagues who  

00:15:33.000 --> 00:15:37.600
we sat together and created these prompts 
in different domains. For instance, we also  

00:15:37.600 --> 00:15:43.560
created social prompts. We also created the same 
kind of graph structures but for reasoning over  

00:15:43.560 --> 00:15:48.480
social structures. For instance, I say, Ashley’s 
friends with Matt. Matt is friends with Michael.  

00:15:48.480 --> 00:15:52.760
If I want to pass a message to Michael, what 
is the path that I can choose? Which would be,  

00:15:52.760 --> 00:15:58.120
I have to tell Ashley. Ashley will tell Matt. Matt 
will tell Michael. This is very similar to another  

00:15:58.120 --> 00:16:04.960
paradigm which was more like a maze, which would 
be similar to saying, there is a castle; it has 16  

00:16:04.960 --> 00:16:10.240
rooms. You enter Room 1. You open the door. It 
opens to Room 2. In Room 2, you open the door,  

00:16:10.240 --> 00:16:16.760
and so on and so forth. So you describe, using 
language, the structure of a social environment  

00:16:16.760 --> 00:16:23.440
or the structure of a spatial environment, and 
then you ask certain questions that have to  

00:16:23.440 --> 00:16:30.240
do with getting from A to B in this social or 
spatial environment from the LLM, or you say,  

00:16:30.240 --> 00:16:35.640
oh, you know, Matt and Michael don’t talk to each 
other anymore. So now in order to pass a message,  

00:16:35.640 --> 00:16:41.280
what should I do? So I need to find a detour. Or, 
for instance, I say, you know, Ashley has become  

00:16:41.280 --> 00:16:46.640
close to Michael now. So now I have a shortcut, 
so I can directly give the message to Ashley,  

00:16:46.640 --> 00:16:51.160
and Ashley can directly give the message to 
Michael. My path to Michael is shorter now.  

00:16:51.160 --> 00:16:56.480
So finding things like detours, shortcuts, or if 
the reward location changes, these are the kinds  

00:16:56.480 --> 00:17:03.880
of changes that, inspired by my own past work 
and inspired by the work of Tolman and others,  

00:17:03.880 --> 00:17:10.320
we implemented in all of our experiments. This led 
to 15 different tasks for every single graph, and  

00:17:10.320 --> 00:17:16.640
we have six graphs total of different complexity 
levels with different graph theoretic features,  

00:17:16.640 --> 00:17:22.280
and [for] each of them, we had three domains. 
We had a spatial domain that was with rooms  

00:17:22.280 --> 00:17:28.160
that had orders like Room 1, Room 2, Room 3; a 
spatial domain that there was no number, there  

00:17:28.160 --> 00:17:33.120
was no ordinal order to the rooms; and a social 
environment where it was the names of different  

00:17:33.120 --> 00:17:40.400
people and so the reasoning was over social, sort 
of, spaces. So you can see this is a very large  

00:17:40.400 --> 00:17:47.840
number of tasks. It’s 6 times 15 times 3, and 
each of the prompts we ran 30 times for different  

00:17:47.840 --> 00:17:53.960
temperatures. Three temperatures: 0, 0.5, and 
1. And for those who are not familiar with this,  

00:17:53.960 --> 00:17:59.560
a temperature of a large language model determines 
how random it will be or how much it will stick to  

00:17:59.560 --> 00:18:07.120
the first or the best option that comes to it 
at the last layer. And so when there are some  

00:18:07.120 --> 00:18:12.800
problems that may be the first obvious answer 
that it finds are not good, perhaps increasing  

00:18:12.800 --> 00:18:17.280
the temperature could help, or perhaps a problem 
that needs precision, increasing the temperature  

00:18:17.280 --> 00:18:22.880
would make it worse. So based on these ideas, we 
also tried it for different temperatures. And we  

00:18:22.880 --> 00:18:29.160
tested eight different language models like this 
in order to systematically evaluate their ability  

00:18:29.160 --> 00:18:36.880
for this multistep reasoning and planning, and 
the framework that we use—we call it CogEval—and  

00:18:36.880 --> 00:18:42.680
CogEval is a framework that’s not just for 
reasoning and multistep planning. Other tasks can  

00:18:42.680 --> 00:18:48.320
be used in this framework in order to be tested, 
as well. And the first step of it is always to  

00:18:48.320 --> 00:18:53.600
operationalize the cognitive capacity in terms of 
many different tasks like I just mentioned. And  

00:18:53.600 --> 00:18:58.080
then the second task is designing the specific 
experiments with different domains like spatial  

00:18:58.080 --> 00:19:04.200
and social; with different structures, like the 
graphs that I told you; and with different kind  

00:19:04.200 --> 00:19:12.040
of repetitions and with different tasks, like 
the detour, shortcut, and the reward revaluation,  

00:19:12.040 --> 00:19:17.000
transition revaluation, and just traversal, all 
the different tasks that I mentioned. And then the  

00:19:17.000 --> 00:19:24.160
third step is to generate many prompts and then 
test them with many repetitions using different  

00:19:24.160 --> 00:19:30.280
temperatures. Why is that? I think something 
that Sam Altman had said is relevant here,  

00:19:30.280 --> 00:19:35.560
which is sometimes with some problems, 
you ask GPT-4 a hundred times, and one  

00:19:35.560 --> 00:19:40.400
out of those hundred, it would give the correct 
answer. Sometimes 30 out of a hundred, it will  

00:19:40.400 --> 00:19:45.080
give the correct answer. You obviously want 
it to give hundred out of hundred the correct  

00:19:45.080 --> 00:19:50.480
answer. But we didn’t want to rely on just one 
try and miss the opportunity to see whether it  

00:19:50.480 --> 00:19:56.720
could give the answer if you probed it again . 
And in all of the eight large language models,  

00:19:56.720 --> 00:20:02.120
we saw that none of the large language models 
was robust to the graph structure. Meaning,  

00:20:02.120 --> 00:20:06.960
its performance got really worse as soon as the 
graph structure, [which] didn’t even have many  

00:20:06.960 --> 00:20:14.920
nodes but just had a tree structure that was six 
or seven nodes, or a six- or seven-node tree was  

00:20:14.920 --> 00:20:20.160
much more difficult for it to solve than a graph 
that had 15 nodes but had a simpler structure that  

00:20:20.160 --> 00:20:26.920
was just two lines. We noted that sometimes, 
counterintuitively, some graph structures  

00:20:26.920 --> 00:20:31.280
that you think should be easy to solve were more 
difficult for them. On the other hand, they were  

00:20:31.280 --> 00:20:37.440
not robust to the task set. So the specific task 
that we tried, whether it was detour, shortcut,  

00:20:37.440 --> 00:20:42.240
or it was reward revaluation or traversal, it 
mattered. For instance, shortcut and detour  

00:20:42.240 --> 00:20:47.160
were very difficult for all of them. Another 
thing that we noticed was that all of them,  

00:20:47.160 --> 00:20:53.560
including GPT-4, hallucinated paths that didn’t 
exist. For instance, there was no door between  

00:20:53.560 --> 00:20:58.160
Room 12 and Room 16. They would hallucinate that 
there is a door, and they would give a response  

00:20:58.160 --> 00:21:04.240
that includes that door. Another kind of failure 
mode that we observed was that they would fail to  

00:21:04.240 --> 00:21:09.000
even find a one-step path. Let’s say between Room 
7 and 8, there is a direct door. We would say,  

00:21:09.000 --> 00:21:15.000
what is the path from 7 and 8? And they would take 
a longer path to go from it. And a final mode that  

00:21:15.000 --> 00:21:19.560
we observed was that they would sometimes fall 
in loops. Even though we would directly ask them  

00:21:19.560 --> 00:21:26.880
to find the shortest path, they would sometimes 
fall into a loop on the way to getting to their  

00:21:26.880 --> 00:21:31.640
destination, which obviously you shouldn’t do 
if you are trying to find the shortest path.  

00:21:31.640 --> 00:21:36.280
That said, there is two differing notions 
of accuracy here. You can have satisficing,  

00:21:36.280 --> 00:21:41.800
which means you get there; you just take a longer 
path. And there is this notion that you cannot  

00:21:41.800 --> 00:21:47.240
get there because you used some imaginary path or 
you did something that didn’t make sense and you,  

00:21:47.240 --> 00:21:52.640
sort of, gave a nonsensical response. We had 
both of those kinds of issues, so we had a lot  

00:21:52.640 --> 00:21:58.640
of issues with giving nonsensical answers, 
repeating the question that we were asking,  

00:21:58.640 --> 00:22:05.400
producing gibberish. So there were numerous kinds 
of challenges. What we did observe was that GPT-4  

00:22:05.400 --> 00:22:14.400
was far better than the other LLMs in this regard, 
at least at the time that we tested it; however,  

00:22:14.400 --> 00:22:23.040
this is obviously on the basis of the particular 
kinds of tasks that we tried. In another study,  

00:22:23.040 --> 00:22:29.520
we tried Tower of Hanoi, which is also a classic 
cognitive science approach to [testing] planning  

00:22:29.520 --> 00:22:35.240
abilities and hierarchical planning abilities. 
And we found that GPT-4 does between zero and  

00:22:35.240 --> 00:22:42.440
10 percent in the three-disk problem and zero 
percent for the four-disk problem. And that is  

00:22:42.440 --> 00:22:49.000
when we started to think about having more 
brain-inspired solutions to improve that  

00:22:49.000 --> 00:22:53.800
approach. But I’m going to leave that for next.
LLORENS: So it sounds like a very extensive set  

00:22:53.800 --> 00:23:00.760
of experiments across many different tasks 
and with many different leading AI models,  

00:23:00.760 --> 00:23:07.520
and you’ve uncovered a lack of robustness across 
some of these different tasks. One curiosity that  

00:23:07.520 --> 00:23:13.800
I have here is how would you assess the relative 
difficulty of these particular tasks for human  

00:23:13.800 --> 00:23:19.440
beings? Would all of these be relatively 
easy for a person to do or not so much? 

00:23:19.440 --> 00:23:24.840
MOMENNEJAD: Great question. So I have conducted 
some of these experiments already and have  

00:23:24.840 --> 00:23:30.520
published them before. Humans do not perform 
symmetrically on all these tasks, for sure;  

00:23:32.080 --> 00:23:38.120
however, for instance, Tower of Hanoi is a problem 
that we know humans can solve. People might have  

00:23:38.120 --> 00:23:45.520
seen this. It’s three little rods. Usually it’s a 
wooden structure, so you have a physical version  

00:23:45.520 --> 00:23:49.480
of it, or you can have a virtual version of it, 
and there are different disks with different  

00:23:49.480 --> 00:23:55.000
colors and sizes. There are some rules. You cannot 
put certain disks on top of others. So there is a  

00:23:55.000 --> 00:24:00.200
particular order in which you can stack the disks. 
Usually what happens is that all the disks are on  

00:24:00.200 --> 00:24:05.280
one side—and when I say a three-disk problem, it 
means you have three total disks. And there is  

00:24:05.280 --> 00:24:11.480
usually a target solution that you are shown, 
and you’re told to get there in a particular  

00:24:11.480 --> 00:24:16.880
number of moves or in a minimum number of moves 
without violating the rules. So in this case,  

00:24:16.880 --> 00:24:23.640
the rules would be that you wouldn’t put certain 
disks on top of others. And based on that, you’re  

00:24:23.640 --> 00:24:30.000
expected to solve the problem. And the performance 
of GPT-4 on Tower of Hanoi three disk is between  

00:24:30.000 --> 00:24:36.840
0 to 10 percent and on Tower of Hanoi four 
disks is zero percent—zero shot. With the help,  

00:24:36.840 --> 00:24:42.680
it can get better. With some support, it gets 
better. So in this regard, it seems like Tower  

00:24:42.680 --> 00:24:48.480
of Hanoi is extremely difficult for GPT-4. It 
doesn’t seem as difficult as it is for GPT-4 for  

00:24:48.480 --> 00:24:56.760
humans. It seems for some reason, that it couldn’t 
even improve itself when we explained the problem  

00:24:56.760 --> 00:25:02.040
even further to it and explain to it what it did 
wrong. Sometimes—if people want to try it out,  

00:25:02.040 --> 00:25:06.080
they should—sometimes, it would argue back and 
say, “No, you’re wrong. I did this right.” Which  

00:25:06.080 --> 00:25:13.600
was a very interesting moment for us with ChatGPT. 
That was the experience that we had for trying it  

00:25:13.600 --> 00:25:21.120
out first without giving it, sort of, more support 
than that, but I can tell you what we did next,  

00:25:21.120 --> 00:25:26.600
but I want to make sure that we cover your other 
questions. But just to wrap this part up, inspired  

00:25:26.600 --> 00:25:32.800
by tasks that have been used for evaluation of 
cognitive capacities such as multistep reasoning  

00:25:32.800 --> 00:25:40.880
and planning in humans, it is possible to evaluate 
cognitive capacities and skills such as multistep  

00:25:40.880 --> 00:25:46.400
reasoning and planning also in large language 
models. And I think that’s the takeaway from this  

00:25:46.400 --> 00:25:53.760
particular study and from this general cognitive 
science–inspired approach. And I would like to say  

00:25:53.760 --> 00:25:59.800
also it is not just human tasks that are useful. 
Tolman’s tasks were done in rodents. A lot of  

00:25:59.800 --> 00:26:07.320
people have done experiments in fruit flies, in 
C. elegans, in worms, in various kinds of other  

00:26:07.320 --> 00:26:15.080
species that are very relevant to testing, as 
well. So I think there is a general possibility of  

00:26:15.080 --> 00:26:23.200
testing particular intelligence skills, evaluating 
it, inspired by experiments and evaluation methods  

00:26:23.200 --> 00:26:29.480
for humans and other biological species.
LLORENS: Let’s explore the way forward  

00:26:29.480 --> 00:26:36.120
for AI from your perspective. You know, 
as you’ve described your recent works,  

00:26:36.120 --> 00:26:44.040
it’s clear that you have, that your work is deeply 
informed by insights from cognitive science,  

00:26:44.040 --> 00:26:50.440
insights from neuroscience, and recent works—your 
recent works—have called for the development,  

00:26:50.440 --> 00:26:56.360
for example, of a prefrontal cortex for AI, and 
I understand this to be the part of the brain  

00:26:56.360 --> 00:27:03.160
that facilitates executive function. How does, how 
does this relate to the, you know, extending the  

00:27:03.160 --> 00:27:09.320
capabilities of AI, a prefrontal cortex for AI?
MOMENNEJAD: Thank you for that question. So let  

00:27:09.320 --> 00:27:17.760
me start by reiterating something I said earlier, 
which is the brain didn’t evolve in a lump. There  

00:27:17.760 --> 00:27:24.160
were different components of brains and nervous 
systems and neurons that evolved at different  

00:27:24.160 --> 00:27:30.160
evolutionary scales. There are some parts of 
the brain that appear in many different species,  

00:27:30.160 --> 00:27:34.720
so they’re robust across many species. And there 
are some parts of the brain that appear in some  

00:27:34.720 --> 00:27:39.840
species that had some particular needs, some 
particular problems they were facing, or some  

00:27:39.840 --> 00:27:47.680
ecological niche. What is, however, in common in 
many of them is that there seems to be some kind  

00:27:47.680 --> 00:27:57.560
of a modular or multicomponent aspect to what we 
call higher cognitive function or what we call  

00:27:57.560 --> 00:28:05.120
executive function. And so the kinds of animals 
that we ascribe some form of executive function  

00:28:05.120 --> 00:28:12.200
of sorts to seem to have brains that have parts or 
modules that do different things. It doesn’t mean  

00:28:12.200 --> 00:28:19.240
that they only do that. It’s not a very extreme 
Fodorian view of modularity. But it is the view  

00:28:19.240 --> 00:28:25.640
that, broadly speaking, when, for instance, we 
observe patients that have damage to a particular  

00:28:25.640 --> 00:28:30.760
part of their prefrontal cortex, it could be that 
they perform the same on an IQ test, but they have  

00:28:30.760 --> 00:28:35.040
problems holding their relationship or their 
jobs. So there are different parts of the brain  

00:28:35.040 --> 00:28:41.840
that selective damage to those areas, because 
of accidents or coma or such, it seems to impair  

00:28:41.840 --> 00:28:48.040
specific cognitive capacities. So this is what 
very much inspired me. I have been investigating  

00:28:48.040 --> 00:28:55.960
the prefrontal cortex for, I guess, 17 years 
now, [LAUGHS] which is a scary number to say. But  

00:28:55.960 --> 00:29:03.000
been ... basically since I started my PhD and even 
during my master’s thesis, I have been focused on  

00:29:03.000 --> 00:29:10.040
the role of the prefrontal cortex in our ability 
for long-term reasoning and planning in not just  

00:29:10.040 --> 00:29:18.000
this moment—long-term, open-ended reasoning and 
planning. Inspired by this work, I thought, OK,  

00:29:18.000 --> 00:29:25.320
if I want to improve GPT-4’s performance on, let’s 
say, Tower of Hanoi, can we get inspired by this  

00:29:25.320 --> 00:29:30.920
kind of multiple roles that different parts of 
the brain play in executive function, specifically  

00:29:30.920 --> 00:29:35.320
different parts of the neocortex and specifically 
different parts of the prefrontal cortex,  

00:29:35.320 --> 00:29:41.360
part of the neocortex, in humans? Can we get 
inspired by some of these main roles that I have  

00:29:41.360 --> 00:29:50.800
studied before and ask GPT-4 to play the role of 
those different parts and solve different parts of  

00:29:50.800 --> 00:29:56.960
the planning and reasoning problem—the multistep 
planning and reasoning problem—using these roles  

00:29:56.960 --> 00:30:04.400
and particular rules of how to iterate over them. 
For instance, there is a part of the brain called  

00:30:04.400 --> 00:30:10.720
anterior cingulate cortex. Among other things, it 
seems to be involved in monitoring for errors and  

00:30:10.720 --> 00:30:16.080
signaling when there is a need to exercise more 
control or move from what people like to call a  

00:30:16.080 --> 00:30:23.120
faster way of thinking to a slower way of thinking 
to solve a particular problem. And there is … so  

00:30:23.120 --> 00:30:28.840
let’s call this the cognitive function of this 
part. Let’s call it the monitor. This is a part of  

00:30:28.840 --> 00:30:34.560
the brain that monitors for when there is a need 
for exercising more control or changing something  

00:30:34.560 --> 00:30:41.360
because there is an error maybe. There is another 
part of the brain and the frontal lobe that is  

00:30:41.360 --> 00:30:46.200
the, for instance, dorsolateral prefrontal 
cortex;; that one is involved in working  

00:30:46.200 --> 00:30:53.880
memory and coming up with, like, simpler plans to 
execute. Then there is a ventromedial prefrontal  

00:30:53.880 --> 00:30:59.080
cortex that is involved in the value of states and 
predicting what is the next state and integrating  

00:30:59.080 --> 00:31:04.760
it with information from other parts of the brain 
to figure out what is the value. So you put all of  

00:31:04.760 --> 00:31:09.960
these things together, you can basically write 
different algorithms that have these different  

00:31:09.960 --> 00:31:16.440
components talking to each other. And we have in 
that paper also, written in a pseudocode style,  

00:31:16.440 --> 00:31:23.120
the different algorithms that are basically akin 
to a tree search, in fact. So there is a part of  

00:31:23.120 --> 00:31:33.600
the role … they’re part of the multicomponent 
or multi-agent realization of a prefrontal  

00:31:34.720 --> 00:31:41.840
cortex-like GPT-4 solution. One part of it 
would propose a plan. The monitor would say,  

00:31:41.840 --> 00:31:46.960
thanks for that; let me pass it on to the part 
that is evaluating what is the outcome of this  

00:31:46.960 --> 00:31:51.920
and what’s the value of that, and get back to 
you. It evaluates there and comes back and says,  

00:31:51.920 --> 00:31:57.120
you know, this is not a good plan; give me another 
one. And in this iteration, sometimes it takes  

00:31:57.120 --> 00:32:05.120
10 iterations; sometimes it takes 20 iterations. 
This kind of council of different types of roles,  

00:32:05.120 --> 00:32:12.560
they come up with a solution that is solving the 
Tower of Hanoi problem. And we managed to bring  

00:32:12.560 --> 00:32:22.080
the performance from 0 to 10 [percent] in GPT-4 
to, I think, about 70—70 percent—in Tower of  

00:32:22.080 --> 00:32:28.200
Hanoi three disks, and OOD, or out-of-distribution 
generalization, without giving any examples of a  

00:32:28.200 --> 00:32:34.800
four disk, it could generalize to above 20 percent 
in four-disk problems. Another impressive thing  

00:32:34.800 --> 00:32:40.120
that happened here—and we tested it on the CogEval 
and the planning tasks from the other experiment,  

00:32:40.120 --> 00:32:47.120
too—was that it brought all of the, sort of, 
hallucinations from about 20 to 30 percent—in  

00:32:47.120 --> 00:32:53.400
some cases, much higher percentages—to 
zero percent. So we had slow thinking;  

00:32:53.400 --> 00:32:58.400
we had 30 iterations, so it took a lot longer. 
And this is, you know, fast and slow thinking.  

00:32:58.400 --> 00:33:02.960
This is very slow thinking. However, we had 
no hallucinations anymore. And hallucination  

00:33:02.960 --> 00:33:11.880
in Tower of Hanoi would be making a move that is 
impossible. For instance, putting in a, kind of,  

00:33:11.880 --> 00:33:18.800
a disk on top of another that you cannot do 
because you violate a rule or taking out a middle  

00:33:18.800 --> 00:33:22.960
disk that you cannot pull out actually. So those 
would be the kinds of hallucinations in Tower of  

00:33:22.960 --> 00:33:28.800
Hanoi. All of those also went to zero. And so 
that is one thing that we have done already,  

00:33:28.800 --> 00:33:33.880
which I have been very excited about.
LLORENS: So you painted a pretty  

00:33:33.880 --> 00:33:41.920
interesting—fascinating, really—picture of a 
multi-agent framework where different instances  

00:33:41.920 --> 00:33:50.760
of an advanced model like GPT-4 would be prompted 
to play the roles of different parts of the brain  

00:33:50.760 --> 00:34:00.440
and, kind of, work together. And so my question is 
a pragmatic one. How do you prompt GPT-4 to play  

00:34:00.440 --> 00:34:04.260
the role of a specific part of the human 
brain? What does that prompt look like? 

00:34:04.260 --> 00:34:10.080
MOMENNEJAD: Great question. I can actually, well, 
we have all of that at the end of our paper,  

00:34:10.080 --> 00:34:17.560
so I can even read some of them if that was of 
interest. But just a quick response to that is  

00:34:17.560 --> 00:34:26.680
you can basically describe the function that you 
want the LLM—in this case GPT-4—to play. You can  

00:34:26.680 --> 00:34:33.960
write that in simple language. You don’t have to 
tell it that this is inspired by the brain. It is  

00:34:33.960 --> 00:34:41.400
completely sufficient to just basically provide 
certain sets of rules in order for it, in order  

00:34:41.400 --> 00:34:51.840
to be able to do that. For instance, after you 
provide the problem, sort of, description … let  

00:34:51.840 --> 00:34:56.560
me see if I can actually read some part of this 
for you. For instance, you give it a problem,  

00:34:56.560 --> 00:35:01.560
and you say, consider this problem. Rule 1: you 
can only move a number if it’s at this and that.  

00:35:01.560 --> 00:35:07.840
You clarify the rules. Here are examples. Here are 
proposed moves. And then you say, for instance,  

00:35:08.360 --> 00:35:18.080
your role is to find whether this particular 
number generated as a solution is accurate.  

00:35:18.080 --> 00:35:24.920
In order to do that, you can call on this other 
function, which is the predictor and evaluator  

00:35:24.920 --> 00:35:31.080
that says, OK, if I do this, what state do I end 
up in, and what is the value of that state? And  

00:35:31.080 --> 00:35:35.560
you get that information, and then based on that 
information, you decide whether the proposed move  

00:35:35.560 --> 00:35:41.560
for this problem is a good move or not. If it 
is, then you pass a message that says, all right,  

00:35:41.560 --> 00:35:46.160
give me the next step of the plan. If it’s not, 
then you say, OK, this is not a good plan; propose  

00:35:46.160 --> 00:35:52.760
another plan. And then the part of, the part that 
plays the role of, “hey, here is the problem. Here  

00:35:52.760 --> 00:35:57.200
are the rules. Propose the first towards the 
subgoal or find the subgoal towards this and  

00:35:57.200 --> 00:36:02.520
propose the next step.” And that one receives 
this feedback from the monitor. And monitor has  

00:36:02.520 --> 00:36:07.920
asked the predictor and evaluator, hey, what 
happens if I do these things and what would  

00:36:07.920 --> 00:36:14.080
be the value of that in order to say, hey, this 
is not a great idea. So in a way this becomes a  

00:36:14.080 --> 00:36:21.120
very simple prefrontal cortex–inspired multi-agent 
system. All of them are within the same … sort of,  

00:36:21.120 --> 00:36:25.680
different calls to GPT-4 but the same instance. 
Just, like, because we were calling it in a code,  

00:36:25.680 --> 00:36:30.720
it’s just, you just call, it’s called multiple 
times and each time with this kind of a very  

00:36:30.720 --> 00:36:38.760
simple in-context learning text that, in text, it 
describes, hey, here’s the kind of problem you’re  

00:36:38.760 --> 00:36:44.640
going to see. Here’s the role I want you to play. 
And here is what other kind of rules you need to  

00:36:44.640 --> 00:36:50.600
call in order to play your role here. And then 
it’s up to the LLM to decide how many times it’s  

00:36:50.600 --> 00:36:55.600
going to call which components in order to solve 
the problem. We don’t decide. We can only decide,  

00:36:55.600 --> 00:37:02.200
hey, cap it at 10 times, for instance, or cap it 
at 30 iterations and then see how it performs. 

00:37:02.200 --> 00:37:04.580
LLORENS: So, Ida, what’s next 
for you and your research? 

00:37:04.580 --> 00:37:13.920
MOMENNEJAD: Thank you for that. I have always 
been interested in understanding minds and  

00:37:13.920 --> 00:37:20.680
making minds, and this has been something that 
I’ve wanted to do since I was a teenager. And I  

00:37:20.680 --> 00:37:26.360
think that my approaches in cognitive neuroscience 
have really helped me to understand minds to the  

00:37:26.360 --> 00:37:35.400
extent that is possible. And my understanding of 
how to make minds comes from basically the work  

00:37:35.400 --> 00:37:45.147
that I’ve done in AI and computer science since my 
undergrad. What I would be interested in is—and I  

00:37:45.147 --> 00:37:50.920
have learned over the years that you cannot think 
about the mind in general when you are trying to  

00:37:50.920 --> 00:37:56.960
isolate some components and building them—is 
that my interest is very much in reasoning and  

00:37:56.960 --> 00:38:04.440
multistep planning, especially in complex problems 
and very long-term problems and how they relate to  

00:38:04.440 --> 00:38:10.600
memory, how the past and the future relate to 
one another. And so something that I would be  

00:38:10.600 --> 00:38:21.880
very interested in is making more efficient types 
of multi-agent brain-inspired AI but also to train  

00:38:21.880 --> 00:38:29.000
smaller large language models, perhaps using 
the process of reasoning in order to improve  

00:38:29.000 --> 00:38:33.760
their reasoning abilities. Because it’s one thing 
to train on outcome and outcome can be inputs and  

00:38:33.760 --> 00:38:38.960
outputs, and that’s the most of the training data 
that LLMs receive. But it’s an entirely different  

00:38:38.960 --> 00:38:44.920
approach to teach the process and probe them on 
different parts of the process as opposed to just  

00:38:44.920 --> 00:38:49.280
the input and output. So I wonder whether 
with that kind of an approach, which would  

00:38:49.280 --> 00:38:54.760
require generating a lot of synthetic data that 
relates to different types of reasoning skills,  

00:38:54.760 --> 00:39:00.560
whether it’s possible to teach LLMs reasoning 
skills, and by reasoning skills, I mean very  

00:39:00.560 --> 00:39:08.600
clearly operationalized—similar to the CogEval 
approach—operationalized, very well-researched,  

00:39:08.600 --> 00:39:14.200
specific cognitive constructs that have construct 
validity and then operationalizing them in terms  

00:39:14.200 --> 00:39:18.760
of many tasks. And something that’s important 
to me is a very important idea and a part of  

00:39:18.760 --> 00:39:24.000
intelligence that maybe I didn’t highlight enough 
in the first part is being able to transfer to  

00:39:24.000 --> 00:39:29.120
tasks that they have never seen before, and they 
can piece together different intelligence skills  

00:39:29.120 --> 00:39:34.040
or reasoning skills in order to solve them. 
Another thing that I have done and I will  

00:39:34.040 --> 00:39:38.760
continue to do is collective intelligence. 
So we talked about multi-agent systems,  

00:39:38.760 --> 00:39:43.560
that they are playing the roles of different parts 
inside one brain. But I’ve also done experiments  

00:39:43.560 --> 00:39:50.080
with multiple humans and how different structures 
of human communication leads to better memory or  

00:39:50.080 --> 00:39:56.400
problem-solving. Humans, also, we invent things; 
we innovate things in cultural accumulation,  

00:39:56.400 --> 00:40:01.880
which requires [building] on a lot of … some 
people do something, I take that outcome,  

00:40:01.880 --> 00:40:05.160
take another outcome, put them together, make 
something. Someone takes my approach and adds  

00:40:05.160 --> 00:40:09.200
something to it; makes something else. So this 
kind of cultural accumulation, we have done some  

00:40:09.200 --> 00:40:13.360
work on that with deep reinforcement learning 
models that share their replay buffer as a way  

00:40:13.360 --> 00:40:19.160
of sharing skill with each other; however, 
as humans become a lot more accustomed to  

00:40:19.160 --> 00:40:25.960
using LLMs and other generative AI, basically 
generative AI would start participating in this  

00:40:25.960 --> 00:40:31.080
kind of cultural accumulation. So the notion of 
collective cognition, collective intelligence,  

00:40:31.080 --> 00:40:36.600
and collective memory will now have to incorporate 
the idea of generative AI being a part of  

00:40:36.600 --> 00:40:44.160
it. And so I’m also interested in different 
approaches to modeling that, understanding that,  

00:40:44.160 --> 00:40:50.480
optimizing that, identifying in what ways it’s 
better. We have found both in humans and in  

00:40:50.480 --> 00:40:55.560
deep reinforcement learning agents, for instance, 
that particular structures of communication that  

00:40:55.560 --> 00:41:00.760
are actually not the most energy-consuming, 
not all-to-all communication, but particular  

00:41:01.680 --> 00:41:06.640
partially connected structures are better for 
innovation than others. And some other structures  

00:41:06.640 --> 00:41:11.280
might be better for memory or collective memory 
converging with each other. So I think it would be  

00:41:11.280 --> 00:41:15.240
very interesting—the same way that we are looking 
at what kind of components talk to each other in  

00:41:15.240 --> 00:41:21.920
one brain to solve certain problems—to think about 
what kind of structures or roles can interact with  

00:41:21.920 --> 00:41:29.080
each other, in what shape and in what frequency of 
communication, in order to solve larger, sort of,  

00:41:29.800 --> 00:41:32.551
cultural accumulation problems.
[MUSIC PLAYS] 

00:41:32.551 --> 00:41:35.680
LLORENS: Well, that’s a compelling vision. 
I really look forward to seeing how far  

00:41:35.680 --> 00:41:38.640
you and the team can take it. And 
thanks for a fascinating discussion. 

00:41:38.640 --> 00:41:47.817
MOMENNEJAD: Thank you so much.
[MUSIC FADES]