Hello, and welcome to the Physics World Weekly
podcast. I'm Hamish Johnston.
This week, we're going to be chatting about
the 2024
Nobel Prize for Physics,
which rather surprisingly
was given for work done in machine learning
and artificial intelligence.
This episode is brought to you by Smaract,
elevating high precision positioning,
metrology, and automation
to empower your breakthroughs.
Shape the future with Smaract Technologies.
On Tuesday,
the Nobel Prize for Physics was awarded to
John Hopfield
of Princeton University
and Geoffrey Hinton of the University of Toronto
for their foundational
discoveries and inventions
that enable machine learning and artificial neural networks.
Needless to say, we didn't see that one
coming here at physics world.
We normally don't consider machine learning and AI
to be disciplines of physics.
And while Hotfield
is a physicist,
Hinton is not.
But we really shouldn't have been surprised.
AI is beginning to change our world,
and some of the mathematics
that it uses
is deeply rooted in physics.
To talk about the prize and the connections
between AI and physics,
I'm joined down the line from California
by Anil
Ananthaswami,
who has written the book, Why Machines Learn,
the Elegant Maths
Behind Modern
AI.
Hi, Anil. Welcome to the podcast.
Hi, Hamish. Thank you very much for having
me. It's my pleasure.
So, Anil, here at, at Physics World, we
were really surprised
by this prize. We were completely caught off
guard if truth be told. Were you surprised?
I was surprised. I woke up, yesterday morning,
and my Twitter feed was a buzz.
Yeah. I I was surprised,
but not necessarily in a bad way. I
think I was pleasantly surprised.
I suppose here at Physics World, we don't
really consider,
AI and machine learning
to be,
to be physics.
But,
of course, Hotfield
is a physicist.
And there's lots of
mathematics
and, and and I suppose,
physics related
mathematics
in,
AI and machine learning. So
maybe it's not that surprising. And and I
would guess that you're, you know I think
I was pleasantly surprised because I thought that,
yes, this is a very important issue in
society at the moment.
And, you know, it's great that the, that
the Nobel Prize Committee is is is focusing
the public's attention
on it.
Yeah. I mean, I think,
you know, when you look at all the
reactions
across the board,
there are certainly people who are upset about
this,
thinking that the standards for awarding the nobles
are a bit inconsistent.
Others are wondering whether the Nobel Committee has
given into the AI hype
and things like that.
And, of course, questions about whether machine learning
really is,
physics belongs,
you know, should we be talking about machine
learning and physics in the same breath and
so on?
I have also seen completely different
and more,
inclusive and positive reactions about this Nobel. There
are people who have very strongly argued that,
physics
plays a very important role,
in machine learning,
as does neuroscience, as does mathematics,
you know,
and computational,
or other computer science. So I think, you
know, you can you can easily imagine the
furor if this prize had been awarded
in medicine
and or neuroscience or something like that.
So my take is that, you know, physics
does,
inform,
machine learning in very,
strong ways. And and the reverse is also
true that machine learning is now playing
an important role in how physics is being
done,
and we can talk about that.
That's right. Yeah. We'll we'll chat about that
a bit later. But, first, I I wanted
to ask you about the winners. Now John
Hopfield is a, a bona fide physicist. He
started out his career in condensed matter physics.
Can can you tell us a bit about
him
and, and and his contributions
to machine learning and AI?
Yeah. Like you said, he started off in
condensed matter physics. And, you know, at some
point in his career in the seventies or
so,
he kind of felt like he had used
up all his particular talents as he called
them, in
in solid state physics. And he was looking
for
new avenues of research, and he actually ended
up moving from
solid state physics and condensed matter physics to,
studying the dynamics of biochemical reactions. And he
made some seminal contributions there. And and the
irony was that he was at Princeton at
the time,
and what he was doing wasn't considered physics
and he ended up moving to Caltech.
And, it was at Caltech that he then
started thinking further about,
how to then take his understanding
of the dynamics of, these,
you know, biochemical,
reaction networks,
to other fields. He was, in particular, very
interested in,
seeing if he could make a contribution to
neuroscience, to computational neuroscience.
And,
and he kept looking for a problem that
he wanted to solve. And eventually, he found
something in neuroscience where
his ideas,
that he had been developing so far could
be applied.
And this was,
the problem of, associative memories, and associative memories
are you know, we all have them.
Imagine you have experienced something,
some some episodic
memory where
this the thing that you experienced had strong
smells or, you know, there was
a song playing in the background and that
thing becomes embedded in your memory. And then
many months later,
a fragment of that smell or a a
a fragment of that song that you heard,
comes into,
your experience and that entire memory is recalled
and and and this is associative memory. And
hopefully realized that
the computational problem of trying to solve,
how to store and recall memories,
could be something he could apply his talents
to, and that was the
beginning of,
his work on Hopfield Networks. He essentially,
designed,
what are today called the current neural networks.
And he showed how,
you know, the principles that from condensed metaphysics,
this are this is the physics of spin
glasses,
could be applied to
train,
such associative
memory networks and recall such memories given a
fragment of the information or given corrupted information.
Yeah. It is. I mean, it is incredible.
The,
yeah, I suppose the connection between
spin glasses, you know, a huge problem in
in condensed matter physics, and,
machine learning and AI.
Despite the fact that Hopfield,
is a physicist, I have to admit, that
I hadn't really heard of him until, yesterday,
or I should say,
Tuesday when the,
when the, when the announcement was made. But
I had heard of Hinton,
who's not a physicist.
And I think that's particularly because he I
think he's become
a bit of a celebrity
even before winning the prize.
He's made some
warnings
about
artificial intelligence and how they could affect
society. So so who is Geoffrey Hinton? And,
and what what has he done,
in the world of AI and machine learning?
Oh, Geoffrey Hinton's fame,
regarding AI,
goes,
beyond just warning us about the dangers of
AI. He he truly is someone who has,
been instrumental in,
you know,
neural network research.
So for instance, if you go back to
the late 19 fifties and early 19 sixties
when the first,
neural networks were
designed and built. These were,
so called perceptrons or single layer neural networks
that were designed by Frank Rosenblatt,
who was a, Cornell University,
psychologist.
Those networks had big limitations.
And, in the 19 sixties,
MIT researchers, Marvin Minsky and Seymour Papert, they
pretty much called poor
rather they
kind of pour cold water on, neural network
research by pointing out that these single layer
neural networks couldn't really solve
very important
or even very simple problems of a certain
kind.
All of that research died, and there were
very few people who persisted and who believed
that neural networks would eventually,
solve the kinds of problems that, they are
actually solving now. And one of those researchers
was Geoff Hinton.
He was doing his PhD,
in Edinburgh,
and, you know, despite his advisors,
lack of interest in neural networks, he persisted.
He kept doing it. And
it was sometime in the mid 19 eighties
that,
Geoffrey Hinton basically with David Jummelhart and,
Williams came up with the paper that showed
how
deep neural networks could be trained using the
back propagation algorithms. They showed how
these networks could learn very important features that
exists,
in data.
And, that was the beginning of the deep
learning revolution.
This particular Nobel,
that Jeff Hinton has been given is for
work that actually predates the back propagation algorithm.
It was for something called Boltzmann's machine.
And,
Boltzmann machines are,
something that follow on from Hopfield's work. So
Hopfield designed Hopfield networks that we were just
talking about.
And Boltzmann machines are further advanced on that,
and,
they use ideas from statistical physics.
So,
honestly,
you know, Jeff Hinton's contributions to AI are
just immense because,
he literally,
was one of the key figures that led
to the deep learning revolution that we are
seeing today.
I see. And
am I right in saying that he resigned
recently
from a position
was it at Google,
because he wanted to speak out
about AI. Can can you talk a bit
about that? What what were some of his
key concerns?
Yes. You're right. I think he did resign
from Google.
He said that he resigned specifically so that
he could talk about the potential dangers of,
AI,
and,
and he's been, he's been talking about that
openly now. He is basically making people aware
that we should be,
you know, paying attention to what's coming down
the line, that if we are not able
to
regulate and control this technology,
this might pose a danger to,
you know, all of us. So he's been
pretty vocal about it, and it's been quite
surprising
how,
you know, people like him and Joshua Benjio
and others have changed their mind over the
last few years.
Because if you go back just 2 years
or so, we weren't hearing such,
fears voiced by,
the main players some of the main players
in AI. Not all of them, of course.
So yeah. I mean, that's something he's been
very concerned about, and he's been very vocal
about in the last,
in a year or so. We've spoken about
spin glasses.
We've spoken about,
Boltzmann
machines. I mean, this sounds like some real
physics here. Can you is it possible for
you to give us a flavor
of of how the concepts of physics
have come into machine learning
and AI and and how they're used?
So John Hopfield basically,
designed his,
Hopfield network,
and based it on the icing model,
of magnetic materials.
And the idea there was that his artificial
neural network had neurons that were bidirectionally
connected.
And
they had dynamics the very in much the
same manner that,
magnetic materials,
when their spins are disturbed, will settle back
into a stable state
and become ferromagnetic
because that particular state represents an energy minimum.
He figured out how to
depict the
energy, quote, unquote energy of
a a recurrent neural network
and
showed how you could store
information
or a memory into these networks
in a way that the stored memory represented
an energy minimum of that network.
And then, if you were to preserve that
network, so which essentially meant you were corrupting
the memory,
the network could dynamically find its way back
to that energy minimum. And that then when
it reached that energy minimum,
you could just read off the outputs of
the neurons
and recover that memory.
So,
it was very, very strongly based on his
understanding of,
you you know, the icing model.
And then so, obviously, physics played a very
large part in the development of Hopfield networks.
And,
Anil, Hinton, you know, as you said, took
took this idea further
using,
using concepts from physics to create this,
Boltzmann machine. And and those are based on,
I'm I'm guessing,
ideas from statistical physics.
Yes. That's absolutely correct. So,
Hopfield networks are deterministic.
And,
what Hinton did was Hinton, along with Terry
Sechnovsky and others,
What he did was,
to
make his networks,
stochastic, and
he did take
strong cues from,
thermodynamics
and Boltzmann distributions.
So, Boltzmann machines are also recurrent neural networks,
which means that the artificial neurons, which are
computational units, are bidirectionally connected.
Recurrent,
the the kind of recurrent neural networks that
Hinton uses, in Boltzmann machines have so called
visible neurons, which are basically the neurons that
you can access, but they also have hidden
neurons.
And but the entire thing is stochastic. And
the whole idea is that
you ought to be able to model probability
distributions
over data.
So you're learning
features or patterns that exist in data in
order to be able to model probability distributions
over data.
And these are, the dynamics of these networks
are designed,
such that the network finds its way,
to some thermodynamic equilibrium. Again, the the use
of the word
thermodynamic
equilibrium or energy state, these are just proxies.
They're obviously, these are pieces of software, so
they don't have,
you know, energy in the same way that
physical systems do. But conceptually, they're operating in
in the same way.
And so Boltzmann machines are essentially
modeling probability distributions over data, and that's a
very important part of generative AI today.
I see. And and so some of the
chatter,
you know, that I I came across
on Tuesday
about this,
the this particular Nobel Prize is that
not only is there lots of physics that's
gone into the development
of machine learning
and, artificial intelligence,
Now physicists are
using
those tools in,
you know, I suppose, a a wide number
of fields from,
developing
new materials
to, analyzing data
from,
particle physics collisions.
Can can you give us a little flavor
of how physicists are using
machine learning and AI in their day to
day work?
Yeah. I mean, we saw today the
chemistry novel going to the
people who figured out the protein folding problem
using machine learning alpha fold, and that's one,
you know, again, that'll be a
point of contention.
But, you know, protein folding has also been
partly a physics problem, and that has been,
you know, solved using machine learning.
There have been numerous attempts
in using machine learning to discover, for instance,
new quantum optics experiments. So given all the
tools
that
experimental,
quantum optics physicist uses on their optical bench,
you can use machine learning algorithms to design
new circuits that
humans may not have discovered because these algorithms
are able to search a very, very large
space of solutions,
and they're optimized to find the best possible
circuits.
For instance,
you know, it's already been shown that such,
machine learning models can,
find better ways to do entanglement swapping,
which may not have been discovered where if
it weren't for these machine learning models.
There have been efforts on there are ongoing
efforts, for instance, to discover symmetries that exist,
in in data for, you know, data that
might be coming out of the LHC.
Of course, once, the machine learning models find
symmetries,
making sense of what these symmetries
actually mean is a much,
larger physics question, and, I don't think AI
at this point, is capable of answering those
questions.
There are also efforts to speed up observation
cosmology. You can you can train machine learning
models
using,
pairs of,
you know, data where on one side, on
the input side are
low resolution
hydrodynamical
simulations, and on the output side are
high resolution versions of the same simulation. So
you you create enough of those,
training data. You train your
machine learning models to
correlate these low resolution,
simulations with the high resolution simulations.
And then once you've trained them, then anytime
you need a new, high risk simulation, all
you have to do is well, I'm saying
all you have to do, but, you know,
with caveats, but you basically,
generate low res,
simulations. And then in fractions of a second,
the machine learning model will give you,
the high res version of that simulation.
If you had tried to do it in
the normal,
sort of computational way, that might have taken
something like a 1000000 CPU hours. So there
are many, many things that are happening,
where machine learning is being is now influencing
the way physics is done. Maybe, even in
condensed matter physics, the the search or in
material science, the search for,
new
materials.
You know, it's very, very hard to,
model create complex models of interatomic interactions.
But if you can train machine learning models
to learn the patterns and learn the interactions,
and then they can be used to predict
new materials with new properties,
all of that stuff is happening. So this
is a two way street. I mean, physics
has
very definitely
influenced,
machine learning in AI and,
you know, and machine learning in AI, I
think, is paying it back now.
We should probably mention one more profound way
in which physics has influenced
AI, which
most listeners would have,
played around with these AIs and these are,
you know, about a year and a half
ago,
we were abuzz with
the image generation models that came out of,
these companies.
For instance, DALL E or Stable Diffusion,
well, those use
very strongly used principles from non equilibrium,
thermodynamics,
and and it would not be possible without
a heavy influence from physics.
I mean, one thing that that I really
find amazing, Anil, is that,
this year, we've got one Nobel Prize
for the development
of a new technology.
And then the next day,
another Nobel Prize goes to
somebody who's used
that technology
to solve a a really, really
important
scientific
problem. I mean, this
this field seems to be moving
so quickly. Is is that something that attracted
you
to, machine learning and AI when you decided
to write your book?
To be honest, I started writing the book
before this hype,
became apparent,
and I don't want to call it hype
in a negative way. Let's you know, there's
so much that has happened in the last
2 years. I started writing this in 2020
when all of this was
completely not on my horizon. I was just
purely
fascinated by,
the mathematics of machine learning. And I really
felt like,
you know, that there's a story to be
told about all of the mathematics that underpins
modern AI.
And as it happened about halfway through the
writing of the book, chat GPT
happened and DALL E and stable diffusion happened
and, you know, things just changed. And the
alpha 4, alpha go, all of these things
have been happening.
And, you know, the last 4 years have
been just insanely,
fast moving.
And I I my suspicion is that we
are still at the very beginnings of this.
I see. And and was it difficult when
you're writing the book?
You know, I'm guessing you you probably learned
new things,
new and exciting things about AI
almost on a daily basis.
Were were you constantly
rewriting
your book or adding new chapters?
I mean, are are you gonna be be
adding a a a chapter now about,
the structure of, proteins, for example?
Well,
I kinda got lucky in the sense that
my focus was on the basic mathematical principles
that underlie,
machine learning. And to be honest,
that that mathematics,
for me at least, in my book, more
or less stops in in the 19 nineties.
So,
it's kind of a historical account of the
math you need to understand what's happening today,
and and
and so that stuff doesn't change.
For instance, we were talking earlier of how
Hinton and Rumelhart and others figured out the
back propagation algorithm. So that's pretty much
the ending of my book. I come to
that, you know, the ability to train large
deep neural networks
is based on this
one single back propagation algorithm.
And it doesn't matter how large the network
is, you know, you could you could train
a network with 1 hidden layer or a
100 hidden layers. The algorithm is same. So
if you understand that, you understand what's happening
today.
I might have to add a chapter or
2 at some point, but, during the course
of the writing of it, I wasn't concerned
with the the rapid changes because
my focus was on being able to tell
the reader that, okay, here are some really
simple,
interesting,
and honestly, in my mind, subjectively speaking, very
elegant mathematics that underpins machine learning.
I see. And so, you you mentioned extra
chapters. I mean, it sounds to me like
you could write,
another book about
exciting applications
of,
artificial intelligence. Do do you have any plans
to do that, or,
do do do you have other interests
these days?
I possibly,
if I were to write something,
it probably won't be about applications. I think
applications, honestly, for me personally,
I I don't find them, that exciting in
terms of storytelling.
So I I I gravitate
much more to,
the basic,
science side of things or the fundamentals.
So I'm still looking for another idea. Some
there are some thoughts in my head, but
nothing has really clarified yet.
I see. Okay. Well, thanks, Anil. Thanks so
much for coming on to the podcast
and, and sharing your thoughts about this year's
Nobel Prize for Physics.
And you can read
much more about the prize on the Physics
World website.
And on the website, you can also find
a review of Anil's book.
Thanks for being on the podcast.
Hamish, thanks you very thank you very much.
This has been a pleasure.
This episode was supported by SmartAct,
empowering breakthroughs in science and technology
with high precision positioning,
metrology,
and automation solutions.
Visit
smartact.com
to learn more.