Hello, and welcome to the Physics World weekly
podcast.
I'm Hamish Johnston.
In this episode,
I'm joined by two physicists
who are exploring
how artificial intelligence
can be used to predict future directions
in quantum science and technology.
They are
Mario Kran, who heads the artificial
scientist lab at Germany's
Max Planck Institute for the Science of Light,
and Felix
Frohnert,
who is doing a PhD
on the intersection
of quantum physics and machine learning
at Leiden University
in The Netherlands.
Earlier this year, Felix,
Mario, and colleagues
published a paper called discovering
emergent connections in quantum physics research
via dynamic word embeddings.
And they join me today to chat about
that work
in which they analyzed more than 66,000
abstracts
from the quantum literature
to see if they could predict
future trends in the field.
Hi, Mario and Felix. Welcome to the podcast.
Hi, Mitch. Very nice, to see you. Glad
to be here.
So, Mario,
can we start off by sort of delving
into the history of this?
Apparently, in the past, people have tried to
use,
data from the scientific literature
to predict the evolution
of a research field. Can you talk a
bit about some of that early work and,
and how successful it was?
Yes.
There have been a lot of, work in
the past where people try to use this
enormous amount of data and see how
researchers work and how researchers could work. So
looking at the strategy of how,
research is done. This has interestingly,
mainly been done by computational
sociologists
who look
at science as a whole and analyze
what
strategies that people use and how we could
accelerate those strategies.
And maybe one of the most exciting work
here in this field that at least,
motivated me to look into this was a
paper from
02/2015,
led by,
Chicago
computational sociologist,
James Evans.
And that paper, it was published in PNAS.
It was called choosing experiments
to accelerate
collective discovery.
And, therefore, the first time, I've seen how
people use millions of scientific papers and patents
to analyze what other concrete strategies that humans
did,
showing that those strategies are at a collective
scale actually quite,
improveable,
let's say, like that. And then they did
a lot of computer simulation to come up
with potential different strategies.
And they have pointed out that if you
would follow if the collective would follow different
strategies, you could theoretically,
come to much faster,
progress in science.
But this was in 02/2015,
and,
this motivated
me at the time that was still during
my PhD to also try the same thing
in my own field in quantum physics. And
then we also,
tried to use hundreds of thousands of papers
to predict,
what scientists will do in the field of
quantum physics. And then we use the same
techniques that James Evans, teams have used. Knowledge
graphs where you have concepts that are the
notes and edges
that are formed when two concepts
are connected,
by a single paper. So you see somehow
the evolution
of,
of what scientists
did in the past.
And that a number of,
there are a number of follow-up papers came.
So this one was published in 2020 in
PNAS, and then we did the AI competition,
but all was based on the same idea
of using this knowledge graphs.
And then,
I met,
Felix,
and Evert,
and they had a very different idea of
how one could
integrate
scientific knowledge and especially the dynamics,
and that is what this specific paper is
about.
I see. Okay. And and I I suppose
it's well, is it obvious? I mean, it
makes sense to me that you've got a
huge amount of data, and that's where machine
learning, artificial intelligence
can really help. And and I think in
the paper that you've published recently,
you've looked at how machine learning can be
used to look for
emergent connections between subfields
in quantum physics with the ultimate goal of
forecasting
the future direction of research.
Can you explain what a subfield is and
and perhaps give an example
of an emergent
connection. I think may maybe a lot of
our listeners would know what a subfield is,
but
the emergent connection bit
might be something new to them.
Felix, could you, could you address that?
Yeah. Sure. Of course. So for us,
subfields in
quantum physics within kind of this broader category
would be something like,
single photon quantum optics. Different sublet would be
gravitational wave physics
or topics related to, I don't know, fault
tolerant quantum algorithms.
All of these would fall under the umbrella
term of quantum physics, but in of itself
deal with very kinds of different questions and
techniques.
While quantum optics might focus on questions like
how to generate,
interfere, or detect single photons,
topics like fault tolerant quantum algorithms
might deal with questions about error correction
or resource estimation,
which, of course, fall under this umbrella term
of, quantum physics but are conceptually quite
distinct. But what we have seen in the
past
is that despite these differences, there can be
fruitful overlaps
between different
subfields
in general in science.
One example that is quite prominent
in the,
kind of physics community has been, I think,
the application of, machine learning to the study
of quantum
phase transitions,
which is a connection that has only been
made in around,
2017,
I think.
Before that, these two
fields
were kind of completely distant. Machine learning at
that time was mostly used for things like
image classification or regression,
and quantum phase transitions were studied with kind
of very
physics specific
tools.
But then in around, 2017,
researchers realized that you could use measurements of,
quantum systems to kind of treat them as
structured data similar to images, making it possible
to kind of combine these two techniques or
apply machine learning methods from computer vision to
physics problems.
And this kind of quite simple connection but
powerful idea,
Combining these two distinct ideas
has led to many fruitful results, basically. And
the goal
for
this research project,
Mario and I did, is to try to
build or kind of improve a data driven
method methods that helps to assess or even
forecast
ideas of kind of that type, basically.
I see. And so, I mean, one thing
that
I can think of in, you know, in
in experimental physics, for example, And I'd like
to see, I mean, have I got the
right idea here when I'm thinking about emergent
connections?
You know, for for example, you've got superconducting
circuits.
And, you know, in the past, superconducting
circuits were were used to make
very sensitive
detectors of magnetic fields, weren't they, squids, which
were used in medical physics and and and
other research. But now
superconducting circuits are used as qubits
in quantum
computing. So
is that the sort of thing that you're
looking at? You know, the the use of
of of one quantum
technology suddenly or well, probably not suddenly, but
slowly
becomes,
useful in another area of physics? So moving
from
sensing
to quantum computing. Is that, I mean, are
those the sort of things that you're trying
to predict in the future, or am I
getting that wrong?
Yes. This would be one of the examples.
There are a number of other
examples that are,
where you would, as a human, potentially not
see the connection,
But just by having
access to hundreds of thousands or millions of
papers, the machine sees that there is some
underlying connection.
And then it can,
say, okay. Very likely, this will be connected
because I've seen this underlying connection before, and
that always leads to connections.
Yeah.
I see. And, I mean, obviously, you know,
being able to predict the future is very
important.
So who, I mean, who would benefit from
this? I mean, it are you thinking of
scientists who are trying to plan their career,
or are you thinking of,
research funders,
looking at, you know, new areas to put
money into or companies
trying to work out what, you know, new
technologies will emerge in the next little while.
I mean, it sounds like it could benefit
a lot of people.
So I think for both,
Mario and me, the motivation in the beginning
was actually just a bit of, self interest
at first. Because as researchers, I think one
of the hardest tasks is to consistently come
up with good and fruitful
research ideas, either for ourselves
or for our students.
Because in order to kind of come up
with many good research ideas, you need to
have a mix of, like, good intuition about
the topic, some experience, and sometimes even just
a bit of luck to come up with,
good ideas.
And the hope for both of us was
to kind of use such a model as
sort of like an artificial
muse in the sense that we can have
a data driven method that helps us to
uncover or assess combination of research topics that
no one has considered before, where there's, like,
no benchmark of, combining these two things
together.
So who would benefit from that? Basically,
anyone working in science who regularly needs to
generate new research ideas or kind of project,
project topics.
Yeah. Good benefit from this tool, I think.
I think the biggest benefit could come from
collaborations,
suggesting collaborations
for
researchers that are not in the same field,
potentially that are in very far away fields.
For instance, let's say I have
a friend who works
in astrophysics. I know nothing about astrophysics.
Now I want to work together.
I would
potentially not even know where to start.
I have my techniques. They have their techniques.
I have no idea,
what is their connection. But the machine has
seen hundreds of thousands or millions of papers,
and they could see some underlying
connection that might,
be possible in the future that has not
been possible,
or that has not been done so far.
So I see a lot of potential for
those
very surprising,
very distant
collaborations.
That would be really great.
I see. And I wanted to ask you
about,
the data that you use to train
your system.
I mean, I'm guessing you that you're using
the titles
of, peer reviewed papers. You're using the abstracts.
Are you also looking at,
references that are made in papers
in order to to make connections? What what
sort of input do you have?
It's actually quite quite simple. So we use
publicly available preprints of quantum physics papers from
the archive.
In general, researchers
submitted their manuscripts to,
websites like the archive before formal, formal publication,
which makes the consent quite easily accessible,
for us. And in particular, we focus on
the abstracts of each preprint in this quantum
physics,
subsection.
The idea behind that is that the abstract
provides,
provides a condensed summary of the kind of
core ideas of the paper, which typically also
includes kind of one or two sentences about
the background,
some information about the main results, and then
also some information about the the impact,
maybe even,
about that paper.
And only focusing on the abstract
and not, for instance, on the kind of
main figures in the paper simplifies the data
quite a bit because, well, typically, in abstracts,
you don't need to deal with,
formulas or citations or references.
And it also kind of, takes away all
the guesswork with things like, figures, for for
instance. Because what abstracts are typically text based,
which means it's,
much easier to kind of extract the information
from that for the machine learning model.
I see. And and is it possible to
explain in simple terms
how the the machine learning is used to
process these data? I mean, you know, sort
of keep in mind that you're you're speaking
to physicists
rather than
computer scientists. Is there
is there an easy way to describe
how it's done? Or may maybe there's not
an easy way. But, I mean, you know,
how would you explain it to a colleague
who's a physicist rather than a computer scientist?
For sure. So in the machine learning pipeline,
there are three main steps. There is a
data collection phase, there is the words embedding
phase, and a training and prediction phase.
The first step, data collection, kind of connects
to the previous question you asked,
where we kind of take a bunch of,
quantum physics publications, look at the abstract,
and then from each abstract, identify
key scientific concepts we are interested in tracking
over time, which in our case are, well,
quantum physics concepts, things like,
entanglement,
quantum circuits, or phase transition.
The second step,
the
word embedding is then the kind of key
idea of this paper. Because in order to
use machine learning to
connect
these concepts,
You need to be able to translate
these words
into numbers that a machine learning model can
understand and process that somehow encodes their meaning
and
usage.
And, yeah, kind of simple ideas like just
giving each word you're interested in a unique
number, of course, wouldn't really work because this
would throw away all the contextual information
about how similar or dissimilar certain words are
or how they are used,
in the context of these abstracts. So what
we do instead is a technique called word
embeddings, which basically turns each word into the
abstract,
into
a high dimensional
vector, which somehow encodes its meaning or how
it is used.
And then you kind of use these,
vectors for the machine learning model. I tried
to come up with a simple example to
kind of highlight
how this could look like. And for this,
I have, like, two sentences.
So the sun is hot
and the moon is cold, which you could
try to encode with such a word embedding.
And in the end, what you would look
for in the embeddings
are then kind of a representation of how
these individual words are used in these two
sentences. So for instance, on some kind of
axis in the embedding, the words sun and
moon might
share one dimension because they're used as the
subjects in the sentences.
The words hot and cold might then lie
might then lie on different axis because they
kind of represent the temperature
of an object.
And the idea is then that somehow the
embedding also encodes
how these different words relate to each other.
So for instance, if you would compute the
overlap between the vector
of the word sun and hot, it should,
if the embedding is well trained, be larger
as then the embedding between sun and cold
because sun and hot are kind of more
closely aligned.
Yeah. This is kind of the the general
idea of how we want to encode the
underlying information about the physics
into these
vectors.
And then the only thing that this,
machine learning model that we then kind of
train to do the predictions in the end
does is look at,
two of these vectors and then try to
predict whether these two vectors
will appear in the same abstract
in a given,
time frame.
What we do in the paper is we
define
a training window, so for instance, from
2000 to 02/2010,
and look at which concept pairs were not
linked yet in this tracking window. And what
the model, tries to do is then to
predict which of these unconnected pairs are likely
to appear together in future abstracts
within a time window of three years, for
instance.
And then the idea is that if you
want some kind of use this machine learning
model for inference to make kind of further
predictions, you basically just shift the time window
from kind of 2,000 to 2,010. You could
also use it to predict from 2,010 to
2,020
or even, further. That's kind of the the
general idea.
I would like to mention
that
what Felix just explained is the
main difference
from our work to what has been done
before.
So before I close what I mentioned in
the beginning, the computational sociologists
and also in my own previous
work, we have built up knowledge graphs. And
knowledge graphs,
have concepts that are the vertices of the
graph, and edges are drawn when,
two concepts are mentioned in the paper.
Now
when I just use the concepts,
I throw away everything about the
context of that word.
That's exactly what Felix mentioned.
Basically, I just define
the word as a number and remove all
of the other information.
But now with the technique that, Felix just
explained that we published in this paper, we
get much more context of the words
by using this more modern,
word embedding instead of knowledge graphs. That comes
with extra technical,
challenges that we had to,
solve,
But then you can get much more context
of the word, of the phrases itself,
and, hopefully,
then get better predictions.
I see. And you use data that went
back to 1994.
So that's, you know, we're looking more
than thirty years ago.
I mean, if you sort of run your
system using older data,
do do you find that it predicts things
that actually happened
a decade or two later?
The short answer to that would be yes.
Well, congratulations.
And there are basically
two different types of predictions our model can
make. Our model can make validation predictions
where we can still
check the outcome and truly
new predictions
where we don't entirely know the outcome yet.
As you mentioned, our model was trained on
data from nineteen nineteen four till,
2017,
in our case, where the model was kind
of used to predict,
contact connections within the next three years, so
until 2019.
And the validation predictions we did
were then
by, kind of shift the time window, as
I explained for the, previous question, where we
tried to predict
or could yeah. Validation predictions,
from 2020
till 2023
where there still is real world data that
we can test against.
And our model managed to kind of discover
a couple of emergent research directions there, which
includes ideas like using tensor network methods to
simulate local quantum circuits, so quantum circuits with
only minimal entanglement,
or ideas of using machine learning to optimize
the structure of,
quantum circuits.
And these kind of, two examples and a
few more are things we discuss in the
paper in detail, basically.
The question about truly
new predictions
is slightly more complicated,
I would say.
One
idea
that I need to discuss for further first
is the idea of,
calibration
of the,
prediction model. Because
in the end, what you are generally interested
in is to have kind of a
small subsets of predictions
where the model is super subbed that,
these predictions
are actually,
correct. So one thing that you need to
check first is
is the confidence
of your machine learning model aligns to the
probability
of being right or false in this prediction?
This is kind of one section in the
paper where we analyze,
like, how our model performs,
in this aspect. And it turns out that
this actually works kind of well. So if
we look at the output probability of the
model, this is aligned quite well with the
probability of the model being false or correct.
So what you can do then is basically
predict
many, many concepts, like, tens of thousands of
concepts,
and then rank them by their probability of
actually occurring
in the future.
And we we did that and found a
subset of predictions that have
over 99.9%
probability
of occurring. And this was very interesting because
most of these concepts
seem to be related to single photon quantum
optics.
So things that are related to concepts like
photon detector and spatial modes
or concepts like polarization entanglements
and,
single photon,
single photon source.
And this was actually something I had a
chat about with with a quantum opt optics
professor from Leiden University, where I'm doing my
PhD.
And all of these predictions actually somehow seem
to make
sense, but are also already
known. So the catch with that is basically
that some of the publications
that were in the kind of subset of
quantum physics,
abstracts that we used
mentioned these concepts. But, of course, we did
not specifically
train our model on optics papers,
which means it's kind of solve these connections,
make these connections,
but it never saw the actual kind of
papers
where these concepts are explicitly mentioned. So there
are kind of
two different interpretations I can make on that.
Kind of one optimistic interpretation would be that
our machine learning model kind of has discovered
the concept of single photon quantum optics on
its own,
and the kind of slightly more conservative,
interpretation would be that in order to get
very meaningful predictions in the future,
one would need to train
our model in kind of all physics applications
to rule out these
kind of trivial or simple predictions
that might have not occurred together in the
training set, but
might be, like, somewhere out there on the
Internet, basically.
I see. And is that is that something
that's possible,
you know, to expand the training
to include all
physics papers? I mean, I mean, first of
all, you know, would you have the computing
resources to do that, and and would that
be useful,
in terms of making predictions?
Maybe you wanna speak about that, Mario?
Yes.
So this is,
definitely possible. Actually, we had, different status where
we not only used,
quantum physics as
input data, but also,
entire physics. Or in one study, we even
used all published
papers,
that exist.
I just looked it up 58,000,000
research papers. So that is possible to take
much, much larger,
subsets
of,
science.
Of course, then
using this data will become
significantly
more computationally expensive
so that the process will be more expensive.
But,
if we really want to go into predicting
novel
research directions, then we have to do this.
So I think in our paper, we showed
it as this huge potential.
And now
we could go into,
thinking about using it for whole physics or
for whole natural science or just whatever
for all science.
Right. Okay. And is that, I mean, is
that something that you and your
collaborators would be interested in? Or do or
do you think that,
that, you know, may maybe maybe people working
or studying other fields, people who study
how biology
evolves, for example, would, would join with you?
I mean, do do do you see this?
I mean, is this a new field
or a growing field that,
that you think you've made a significant
contribution to,
and, you know, other people will follow on
from you?
There
there are, there are a lot of groups
that look into
this direction or related direction. It's a huge,
field to,
to try to come up with
computer inspired new ideas in a, data driven
way.
We show one method how that is done.
There are many other methods.
Now the question is,
is
predicting what scientists will do,
is this the one thing you need to
discover new ideas or are there other things?
For instance, you can think about, can I
predict
impactful
research directions?
And just a few months ago, we have
also published in machine learning science and technology
and,
IOP,
journal,
a paper where we showed,
that was, led by,
my former post doc, Shumei Gu, who is
also coauthor
of the paper with Felix,
that we showed that you can actually predict
also,
impactful research directions.
So let's say you have two concepts
that have never been started before together.
Now the question is,
will there be papers
that started it? And the next question is,
will those paper that studied this
topics that have never been studied before, will
they be impactful?
Will they,
create a lot of citations?
And very surprisingly
to us, this is also predictable.
So now we can also predict in some
way what future research directions could be impactful.
Of course, what Felix has done, what is
new, and what could, could be done and
what might be done. And then I think
the final question is what research directions
are interesting
for humans?
For that,
one could do,
large scale evaluations,
using humans.
We have started,
doing this, and I think there are thousand
different things one can do.
One thing I've seen
in this whole,
this whole range of work is the humans
are exceptionally good in coming up with ideas,
and it's really not clear where those ideas
come from.
I think there's a lot of a lot
of things that we need to understand.
From the human perspective, why are human create
scientists? Where are those ideas coming from? That
we can artificially recreate this.
I see.
And and what about, you know, this specific
project,
that that you, Felix, and, Mario are working
on?
Are you are are you following up this
research? Are are you planning on improving
your machine learning
technique,
or maybe applying it to more data? I
suppose well, you have applied it to more
data. What what what's next for you? Or
are you moving on to something else?
So there are a couple of kind of,
obvious follow-up research directions that,
one could pursue.
There are,
a couple of ideas about how to improve
the embedding that we came up with,
even further.
One of these ideas would be that
the kind of main application of our
embedding was to kind of replace the information
in the knowledge graph as Mavi
mentioned earlier. But
you don't necessarily
need to view it in this kind of
binary setting of either only using the knowledge
graph or or embedding. You could also come
up with some kind of combined representation of
both that,
takes kind of as much information as possible
and then just kind of gives it to
the machine learning model and lets it do
its thing,
basically.
The other,
obvious direction would be to,
apply our methods
to other fields.
Like, on the archive alone, there are
more than 15 other primary categories beyond quantum
physics that we could try to use this
method on where the data is basically
publicly available and on the Internet ready to
use. This includes topics like, mathematics, computer science,
or economics or biology where we could also
try our method on and see if we
can find emergent research connections in these fields.
And Mario, what about you? What are you
looking forward to in the future?
Yeah. So beyond things like predicting,
more general properties like cetaceans
and,
maybe surprise and so on,
one
could one could probably make big steps into
improving their AI models themselves
by performing AI competitions.
So I think the data that Felix,
has is very well equipped for that, where
you would have a pool of prize money
and then a hidden dataset, and participants need
to provide
the best
models that they can come up with to
predict
for predict, for instance, the future of, quantum
physics.
And then test in that case, we could
compete,
techniques like knowledge craft, the techniques that we
have published in these papers,
with many, many other techniques that other people
come up with. And that is, I've seen
this, now one time,
very clearly that this is very useful. There
are many, many diverse techniques that usually
be in such competition,
techniques that you alone would never think about.
It's
significantly more technical,
ideas, significantly
simpler ideas that you would not even believe
that they might work.
So that would be really great. And I
think this
type of data that we,
prepared here would suit very well for such
AI competitions.
I see. And and Mario,
you you mentioned the journal, Machine Learning
Science and Technology.
Can you and you're on the editorial board
of that journal. Can
you just give us a little bit of
a flavor of what sort of physics related
research the journal publishes?
Yes. So this is
probably one of the most important, if not
the most important,
journal in that
field. Because
machine learning research,
getting it published in physics journals,
can be
quite challenging because,
physicists are looking at different things that we
might look at. Also publishing
those,
techniques
in AI,
conferences might also be very challenging because
those people also look at very different things.
So there was before this channel, there was
really a gap where it's not clear how
would you even publish such works. And now
I think,
MLST became
the
the two go
address
for exactly,
such works.
And then a lot of works on how
you would,
use new
ideas for machine learning in
physics and chemistry and biology.
Our work is a little bit, outlier because
it even goes a step further to kind
of meet the science, but you see that
MLC is, quite open in,
disrespect.
So,
that's why I'm super happy that the channel
exists and also super happy that they invited
me to join the, editorial board.
Well, that's great. Thanks. Thanks, Felix, and Mario
as well. Thanks for coming on the podcast
and talking about your research. It's,
it's a fascinating
application of, machine learning,
and artificial intelligence.
And, yeah, we hope to, to learn more
about it from you and your colleagues. Thank
you.
Thank you very much. Thank you. Bye.
That was Felix Frohnert of the University of
Leiden and Mario Krenn of the Max Planck
Institute for the Science of Light.
Thanks to both of them for a fascinating
discussion.
You can find their open access paper in
the journal,
Machine Learning,
Science and Technology,
which can be found on the IOP Science
website.
Just look for the title,
Discovering
Emergent Connections
in Quantum Physics Research
via Dynamic Word Embeddings.
I'm afraid that's all the time we have
for this week's podcast.
I'll sign off with a thanks to our
producer, Fred Ailes, and an invitation for you
to join us next week.