Physics world.
Hello and welcome to the Physics World Stories Podcast. I'm Andrew g Tron.
In this episode,
we're gonna be exploring Moore's Law named after the late Gordon Moore,
who observed that the number of transistors in a dense integrated circuit
doubled about every two years of late.
There have been suggestions that the law might be dead.
We'll explore that and look into why this matters and what,
if anything, might replace it. Later in the podcast,
we'll hear from two researchers looking into the future of computing.
Two of the biggest questions that they're looking into are whether the increase
of computing power can keep up with the ever increasing demands of generative
artificial intelligence,
and can that happen in a way that doesn't threaten the planet due to the vast
energy demands? But first, here's Louis Barson.
I am, uh, director of Science,
innovation and Skills at the Institute of Physics.
Semiconductors are important because they're the basic, uh,
building blocks of computer chips. Um,
and the more transistors you have on a chip, uh,
the more numbers you can crunch, uh,
the smaller and more powerful your digital devices become. Uh, and, uh,
that's important. Cause that's what really powers the, the digital revolution.
It's the miniaturization, uh, and the increasing, uh,
power and efficiency of digital devices that is transforming industries,
uh, in and everything from aerospace and automotive to, uh, healthcare, uh,
and defense. And I guess it's Moore's Law, um,
which tells the story of this doubling, uh, of, of that,
that capacity that really charts the development of the digital transformation
that we're seeing.
Can you tell me a bit more about Moore's law and crucially, is it actually dead?
I guess this really comes from a,
an article he published in 1965 in the American Journal, uh, electronics.
Uh, and this is back when he was director of the R&d Lab at Fairchild's
Semiconductors, so before Intel and all that. Um, and the,
the article and the prediction is about the number of components on a chip.
So it's a pretty understated article.
I would say It's sort of highlights that the,
the number of the components on a chip had doubled every year for the last three
years. And he said, you know, he was, he was open and said that, you know,
there's a bit of uncertainty of the future growth rates,
but he expected that doubling would continue every year,
at least for 10 more years. Um, so he then updated his prediction in 1975.
Actually,
to save this doubling would take place every two years rather than every year.
So this was, this was the observation of a trend.
It was charting it into the midterm and, uh, you know,
updating the prediction to make it a bit more realistic, but not,
not a prophecy by, uh, by stretching the imagination. Uh,
and I guess it was probably actually, uh,
the eminent Caltech professor Carver Mead,
who was a close colleague of Gordon Moore. Uh, he also had a,
a good knack for predict in the future. Um, so for instance, you know,
very early on when,
when many thought miniaturization would actually make chip performance worse,
he did some clever physics and material science to show that in fact,
miniaturizing ships would allow them to get faster and more efficient.
And that turned out to be the case.
So I think it's actually Carver Meat who's credited as the, um, uh,
the one who actually referred to Moore's prediction as Moore's Law. Uh,
and that raised, raised its profile and highlighted its significance. Um,
and you knows partly through that pro popularization,
we start to see actually Moore's Law becoming a rallying call for the industry.
Uh, a bit of a target for, uh, chip companies to aspire to, uh,
as well as just an observation of what's happening. So it,
it developed a little bit of a felt self-fulfilling nature, um,
in that the companies themselves were actually trying to, to, to, to reach it.
Um, but you asked about whether, whether it's dead. Um, it's,
it's been amazing how long it's lasted, right? So right at the beginning,
I think people were talking about, you know, 10 years, uh,
perhaps it could last for as long as 10 years.
We've seen that that doubling trend over every two years has ca actually carried
on, uh, from the early 1970s, uh, to, to the, the 2020s.
So we've seen, um, you know,
the numbers of transitions on ship that go from somewhere around 5,000 in,
in the 1970s to more like 50 billion in the 2020s. Through that,
that success of doubling every two years,
it's really mind blowing that improvement curve could,
could stay at that expon level for 50 years. Um,
uh, so I think we, we are starting to see it level off. Um,
but it's been amazing, uh, what it's achieved, uh, over the last 50 years.
And actually that leveling off potentially from a UK perspective is really
interesting. Um,
because that means perhaps there are new technologies that need to be looked at
that could, um, could help to continue the curve,
help continue that transformation. And perhaps that's a chance for the UK to,
um, uh, to really articulate the,
the role it wants to play in this industry moving forward.
The UK government have, uh, to some extent, at least,
said what they would like to do.
You know, it's clear how transformational this this industry is. Um, uh,
you know, access to chips is now a matter of critical economic importance. Uh,
it's not just something that's important for consumer electronics.
Every industry from medical devices to defense and security to aerospace and
automotive have critical reliance on access to cutting edge chips.
And I think we saw during the pandemic that, uh, you know,
a worldwide chip shortage actually caused the automotive supply chain issues
that led to more than a 25% slump in auto production and some big backlogs.
They're only just getting unwound. Uh,
and it's worth noting that chips themselves.
So now one of the world's biggest markets,
roughly half a trillion dollar industry, um, and, uh,
semiconductor chips and other world's fourth most traded products after crude
oil, refined oil and cars, it's chips. So, uh, you know,
it's not surprising that, that sometimes people refer to them as like the,
the new oil. They are very nearly as traded and very nearly as vital, uh,
for other worlds major industries.
So it's not really surprising that governments are taking notice. Um, you know,
as, as you and others will know, the us, China,
and Europe are slugging it out in a super high stakes battle of supremacy where,
um, you know, building a new fabrication facilities often in the,
the billions of dollars,
I think tsm C'S latest plant in Arizona triples their overall investment in the
state of 40 billion. So it's, it's really eye watering stakes. Um, I,
I guess for the US government, for instance,
recently committed to more than 50 billion to support industry and innovation in
its chips act, um, last year. So, uh, you mentioned the UK sort of setting out,
its still, um, the, the context is that right? The UK is a,
a mid-sized economy. It can't invest, uh, at the levels of the us, China,
Europe, but it has a lot of strengths in chip design and packaging and some next
generation technologies like, uh, compound semiconductors, 2D materials, but it,
it can't just go toe to toe with these larger economies. So it had, uh,
a real challenge. Um, and, uh, I guess,
uh, it's worth just talking about the, um, the context of ARM as well.
I think that was, that was really important in the, the genesis of the strategy.
So, um, I guess the UK's flagship design,
the chip design company arm, which came out of the, um,
the nineties investment in computing around Cambridge, uh,
some of you might remember the Acorn Edes had a bit of a su suspect with a
child. So Acorn essentially developed into Arm, uh,
became one of the world's leading chip design specialists with their chips in
95% of the world's smartphones, for instance. And in, uh, in 2014,
they were purchased by, uh, the Japanese tech and venture capital company, uh,
SoftBank.
Just a quick aside from me,
because that mention of Acorn computers took me on a trip down memory lane,
and I discovered that the computer that I was a particularly proud owner of in
1983, the Acorn Electron,
was the biggest selling computer at the time in the uk,
but it only sold between 200 and 250,000.
And it just made me wonder is that anybody listening to this podcast who also
had an acorn electron, if so, perhaps you could tweet us at Physics world.
But anyway, back to Louis Barson.
And this really illustrates the dilemma that we have here in the uk.
So arm sold for I think 32 billion, which is, you know,
a lot of money and is clearly, uh, clearly a success by many measures. But,
you know, some,
some felt and still feel that it may not have valued its national significance
appropriately and may have actually weakened the, the domestic ecosystem. Um,
so this, this was a big debate, uh, and after this,
the government ran a big review of its strategy around critical technologies,
um, which led to, uh, a new more strategic approach to, to,
to national critical technologies and to the commitment to a semiconductor
strategy. Um, so this felt like a big opportunity for the UK to start to think,
uh, you know,
a bit more strategically about the role he wants to play and to carve out a bit
of a niche. Um, so that's where the IOP recently ran,
uh, an impact project in partnership with, uh,
the Royal Academy of Engineering to,
to support the UK government in thinking about the position they wanted to take,
uh, towards this, you know, critical technology. Uh,
and we got together an expert round table who produced, I think some really,
you know, insightful recommendations that helped inform the evidence base. Um,
and, uh, you know, when the strategy came out earlier this year, uh,
we gave it a cautious welcome. And I think that was firstly because, you know,
it was really important to have the strategy itself, uh,
for the industry to hear the strategic significance that, you know,
the government places on this technology for the world to see the value replaced
in the industry and, and the plans that we have. Second, cuz it felt, you know,
really well structured, it was, uh,
it was good to have the focus on growth in the domestic industry as well as,
you know, a recognition of the importance of, uh,
security of supply of chips for the industrial sectors that are big users from
healthcare to water. Uh, and, you know, looking at the, uh,
the national security dimensions, uh,
so not just from the perspective of security of, for instance,
wireless infrastructure, you know, 5G infrastructure,
that's been such a big thing, but also, you know, the vital role of,
of chips in, in defense and security. Um, so it seemed,
it seemed well structured and it seemed to recognize this narrative of this
being a chance for us to, uh, to position, to, to dive in. Uh,
and third, uh,
cuz it seemed to take on board the I o P round table recommendations, uh,
really, really well, uh, including, uh, a recommendation around, uh,
having in national infrastructure initiative and, uh, you know,
the joined up skills focus. So, um, so that was all good. Uh,
should also say, I guess it was, you know, it was a relatively modest strategy.
I guess the, the scale of the strategy was, was 1 billion over 10 years. Um,
and we need to recognize in the, in the,
in the context of worldwide semiconductor investment, that's not huge,
but it felt like a really important starting point and statement of intent. So,
um, you know, really I think this gives, gives the UK the chance to, um,
be using the ambition and the momentum and the strategy to prepare to,
to build on some of our strengths and,
and develop the semiconductor technologies of tomorrow so that when the, uh,
inflection point for, uh, you know,
Moore's law slowing for silicon versus other technologies comes,
we are ready to join the race for the headstart and that we've got the,
the stomach to invest at the scales that will be needed when the time comes.
Th this 1 billion pounds over 10 years,
is that the sort of timescale where we'll need to be investing more before those
10 years are out?
I think, I think almost certainly is the, uh, is the answer, the honest answer.
I mean, this is a starting point and, you know, over, over the next decade, uh,
we're gonna see quite a lot of, um, uh, of development, I think,
in the semiconductor industry. So, um,
I think we're almost certainly gonna be hearing more about what the,
the government thinks about semiconductors and for really this,
this strategic direction to just be a starting point. Um,
and it's really good news. I think they've established a uk, uh,
industrial advisory council to help guide the development and the evolution of
the strategy. Uh, I know, um,
the people I've spoken to and the team are very open to, uh,
the idea that actually, you know,
we may need to think again and actually inject more ambition of the strategy at,
at the moment when it becomes clear, you know, what the,
the next step should be for the uk.
Should the physics community be interested in this and should everybody else be
interested in this?
Uh, yes and yes, I think is the answer, ,
what Moore's role really shows is how central cutting edge physics and material
science is to, uh, the digital revolution.
So it shows that, uh, through reducing,
through clever physics and material science through miniaturizing chips to
increasing the number of transistors, increasing the number of, you know,
ones and zeros you can represent on a, on a chip. Um, you, uh,
increase computing power and you reduce the cost of computing technology.
Um, so that really shows that, you know, it's actually physics and, uh,
a material science right at the heart of the digital revolution. Um, uh,
Moore's law really charts the contribution, I would say, of,
of physics and material science to, um, to, to the digital, technological,
technological revolution. And, um, I think, you know,
beyond the physics community, obviously, uh, everyone, uh,
it should be interested because, uh, this is, this is transforming, um, the,
the way in which we, we live and work,
everyone now has a smartphone in their pocket. Uh, in the 1965 when,
when Gordon Moore coined the term, uh, nobody, uh,
had a smartphone in the pocket that was not even in the realms of something that
could be thought about. Um, uh, cuz transistors, you know, you'd,
you'd hold them in your hand and they'd be, uh, uh, you know, a valve or a,
in a, a, you know, in a light bulb shaped, uh, uh, uh, apparatus. So,
um, I think, you know, if, if, if you care about the revolution,
if you care about the direction of transformation in sectors, you should,
you should care about, uh, uh, Moore law and semiconductors. Um,
and that's true from a, uh,
those in the physics community who care about these things, but also, you know,
those in, in wider, uh, society who care about these things.
Whether Moore's law is dead on life support or otherwise, the problem is,
whilst making the transistor smaller means that the computers are faster,
there is a limit to how small and how many you can fit
on a chip.
In, in simple terms, uh, you know,
transistor is a switch that can be turned on or off, so it can represent,
you know, the one or zero of, of binary code, which is the,
the basic language that can be to speak. So, uh, the more transistors you have,
the more ones and zeros you can represent, uh,
and the more complex calculations you can do more quickly. Um,
so, so shrinking those transistors means you get more on a chip and each chip
becomes able to do, to do more. Uh, but you're absolutely right.
We're starting to get, uh, the, the, the individual elements on a,
on a chip are starting to get so small. Um, you know, we get,
we're sub 10 nanometer now. I think the,
the latest process is ar around three nanometers, um, which is, you know,
getting to the, uh, the scale of individual atoms. Um,
you start to see quantum effects coming into play that you don't,
things that we don't see happening in everyday life happen in chips because
they're so small. Um, and, uh, we are definitely beginning to,
to, uh, reach, uh,
the limits because I suppose if this continues on for another, uh, 50 years,
we're gonna be, you know, individual elements would have to be smaller than, uh,
individual atoms, which, uh, is clearly, clearly not feasible. Um,
but there are interesting technologies that are starting to extend the
boundaries of what that could mean. So, um, for instance,
3D chips that use more complex chip architectures to mean that, uh,
you can squeeze, uh, more, more life out of Moore's law, uh, are starting to,
to become more popular. Um, and I mentioned a couple of, uh, potential sort of,
um, UK strength opportunities previously, but, uh,
compound semiconductors are a great example of, um, a new material science,
physics technology that, um, um, is using multiple elements, not, um,
uh, not just the traditional silicon to create substrates for,
for chips that could be more efficient and effective. Um,
2D materials such as graphene and others, um, have some potential if,
if we can manufacture them in, in ways that give them, uh,
semiconductor characteristics. Um,
and there's also interesting experiments around using, um,
photonics or using lights to, uh,
to do computing in ways that could be more efficient and effective. So for me,
it's exciting that Moore's law is starting to slow because we're,
we're reaching the limits of that,
that way of looking at semiconductors and starting to,
it's starting to mean we need to look at other ways of doing it, which, uh,
to me is very exciting. And to,
I know to others in the physics and and engineering community is very exciting
as well.
One such person is Thomas Verda.
Delma. I'm a researcher at NEC Labs America, uh,
based in Princeton, New Jersey.
And my research field is on the topic of optical computing or sort of
next generation computer hardware with using optics optical
computers. Uh,
they rely on the physics of, um,
light waves in order to perform some computations.
So traditional electronic computers are based on,
um,
transer gate and information is encoded into a voltage
that if it's above a certain threshold, then there's a one, if it's below,
it's a zero. So, very binary, very efficient way of, uh,
connecting the device to binary logic. Uh,
on electronics in optics, what you try to do is do the same thing,
uh,
but instead of storing information voltage you store in some property of light,
like it's amplitude or it's color, or it's polarization.
And so the challenge here is that there is no equivalent
to a transistor in optics, right?
So what I,
what I've researched when I did my PhD at Princeton University was,
how do we think about a computer where we cannot have an optical transistor,
right? And in order to take it full advantage of, um,
light properties for computing,
what we needed to do is to use linear computations with optics and non-linear
computations with some el opto electronic device.
And the architecture that most resembled what these
advantages were neuro networks, coincidentally, right?
Which were on vogue, uh, since 2009,
um, with, with machine learning. So optical computing,
you'll hear from different researchers, they come in multiple flavors.
A lot of people focus on the linear algebra part of it by optimizing, uh,
sort of matrix,
vector multiplication and reducing the energy consumption of that.
So creating like, uh, accelerator chips, um, that focus on,
on this aspect of computation.
Some others would look into image processing for convolution
operators. And that there, you wanna do, uh,
you wanna process as many images as possible, as fast as possible,
things like that. But in all the situations, what I see is they,
they're gonna act as, um, accelerator processors.
They need to work in conjunction with, um, um,
state-of-the-art electronics processors.
So you shouldn't think of optical computing as an evolution or an iteration
of the, the history of computers.
You should think of a new technology that will enable, um, um,
you know, accelerator functions in, uh,
in for multiple tasks that society will need in the future. That's,
that's my opinion at least.
But we'll still need other types of computers.
Well, well, there's certainly a need,
but to me it's unclear what's gonna be, um, if, if,
if ever there's gonna be, uh,
a new paradigm that will replace the current, uh, sort of, uh,
digital electronic, uh, computers.
So people talk about, uh, quantum computing, for example,
um, or, uh, using more advanced, um,
physics like spintronics and things like that. So, um,
but so far as I know, as I've seen,
I don't think none of those technologies will replace fully the kinds of, uh,
algorithms that current computers can run.
Okay? But does that mean that, I mean,
should we care really if Moore's law ends?
Yes. Yes. Uh, it, it has a lot of consequences in my opinion.
Um, the, uh, the reason why we should care is because
now we are entering a new period of, um, machine intelligence.
And so the amount of computation demand required by
society is now going to scale, uh,
beyond the individual human demands, right?
So now we have machine to machine sort of communication processes.
We'll have automated systems,
we'll have self-driving cars and things like that.
So machines, we can build more, many more machines than the human population.
So there's no upper upper limit on that. So as,
as the men for computation for this, uh, AI models, for example, with the,
uh, Chad g p t, as they increase,
we're gonna need more computing power. And with the end of Moore's law, what,
you know, what we were used to in the past was, okay,
we just wait two years and we gain an, uh, uh, an immediate,
you know, two x efficiency after two years, right? Because of more slow.
Now that's not true anymore. If you need more computing,
you're gonna have to spend, um, more money on chips, pay,
pay more for transistors, and you gotta, it's gonna consume more, uh,
electricity, right?
So we're entering this new regime from abundance to,
to scarcity, right? It's gonna become like, um,
a commodity like oil, the chips as you, as you know, as you probably heard on,
uh, on the news. Yeah. So, so I think in that context,
that's why more Moore's law ending is, is important.
You are pretty certain from what you're saying, that Moore's law is dead.
If it's not dead, it certainly has slowed down. And so that's, uh,
very visible from the, um, the,
the sort of,
the number of transistors on every chip as that they evolve has,
has not been, um, following on the same trajectory as it has, uh,
as it had been since the 1970s. So I think so,
but more importantly than more slow, there is another, um,
scaling law, so to speak, that's called, that I pay more attention to,
which is called the Denars law. So the Denars law,
it's more like a scaling law,
is the sense that as the dimensions of a device go down,
so those power consumption,
so smaller devices within the dielectrics and shorter channels, uh, um,
they improve the, um, the frequency, the cost,
and the power consumption of the device. So it, it sort of had this, uh,
virtual cycle of more slow in parallel going on where
the chips as you, you packed more transistors on chip by making them smaller,
they also became more efficient. You see?
And that stopped around 2005, and that's when you saw your, you know,
your computer CPUs, if you paid attention, the frequency they operated at. Um,
and they had been growing like doubling every couple of years,
400 megahertz,
800 megahertz until they stopped at about three to four
gigahertz, right? And then since then, instead of improving the frequency, uh,
chip manufacturers started adding more cores, so to speak.
So how many cores does your Intel CPU has, right? One core,
two cores, four, eight, right? Well, at the same clock frequency.
So instead of, um, reducing the size and using a single core,
they are increasing the size of the chip using more transducers that way.
But more cores, you have more power consumption, right? And that,
that's the, the direct effect of the end of the des then they're scaling low,
right?
So that's something that is also interesting to think about beyond the more,
more slow.
Do, do. How do you feel about the future? Because if you look at the news, um,
reports, there's great concern about ai, for example,
that maybe there's great concern about, um, less,
with less noise around it, this concern about, um,
transistors becoming a commodity than the same way as oil.
What's your feeling on.
It? I'm a hardware researcher.
I usually concern myself with thinking about the problems related to,
uh, uh, power consumption.
How can we actually support this demand? How can we, uh,
improve the efficiency of computing so that it's, uh, um,
it doesn't contribute even more to, to the, uh, to climate change?
So, but yes, it, it concerns me on a, you know, citizen level,
uh, the, uh, development of the, the, this,
the high speed development of AI models and how they're actually becoming more
integrated in society. I'm not, um, I'm,
I'm an optimist though on that, on that front.
I do think that on a personal opinion,
that the, um, the, uh,
the AI models that we have seen in
recent years, in particular, the, uh, the G P T three and four,
they are getting close to, um,
to, they're getting close to hitting the limitations of hardware, right?
They're already too expensive to train on the millions of dollars.
So the question is, if they want to increase, does that,
that model by tenfold,
are they prepared to spend 10 times more money, right?
So I think that's gonna be a real bottleneck in the, in the short term.
And so hardware engineers like me are trying to catch up to that trend,
right? Um, not not just me, but of course the whole industry. And,
um, so if we don't catch up,
and if that trend continues to increase, then, um,
right now people don't pay attention to this,
but data centers already consumed 3% of the global electricity.
So if you multiply that by 10, that becomes 30%, right?
And, uh, 3% is tolerable, 30% isn't. Okay.
So between now and then, then we have to
really solve this problem one way or another.
That's mainly what I concern myself with so far.
When is this time when we're gonna have to solve this problem?
Um, it, it depends. It depends on the, uh, appetite for, you know,
machine learning models to keep increasing, right? So in recent past,
because it was unbounded, the, uh, training models were doubling,
uh, in size in, in a timeframe of less than a year,
I think every year in model doubled. But, so if you continue that,
a factor of 10 is just three years away or four years away, right? Um,
but it's not gonna continue like that, at least not in the short term.
So I'm thinking, I'm thinking a decade. Uh, that's our,
that's what I have in my mind in a decade. We should have, um,
more efficient accelerators for, uh,
this kind of computing that machines require. Um,
and also working in parallel with that, we have a lot of innovations from the,
uh, software, uh, from the software side with,
uh, better mathematics,
better ways of representing information digitally,
better neur network models. So let's not forget that the, uh,
transformer model is only a few years old, and, you know, what's,
that could be a new model that will power the next generation, uh,
sort of AI model in which it's gonna be way more efficient, for example.
I could see that coming as well.
Yeah. What would you like to see happen? In an ideal world?
I think computers, um, I,
I kind of partake Steve Jobs vision for this.
It's supposed to be a tool to empower humans and
human communication, human uh, productivity.
And so I do like the sort of the,
um, the direction that it has taken so far.
This AI models are helping, um, a lot of people,
particularly knowledge workers, right?
So I would like that to move towards,
um, the, um, cyber physical space.
So instead of having your AI help you draft a
better text or par your code or things like that,
it'd be nice if you could have AI help you with your, uh, physical tasks.
Maybe you are fixing your car and you can
diagnose it for you, and it can help you figure out which, you know,
which knobs to, to turn, which screws to tighten, things like that, right?
Um, so more, more cyber physical systems. I think that's, uh,
that's what I would like, uh, to see.
Another situation is, uh, think about, um,
human machine interaction, right? Being more, having a higher bandwidth,
uh, of communication between yourself and the machine, rather than right now,
we, we can speak to it, we can type, can we think of something better than that?
So that's something that, uh, that company, uh, Elon Musk's, uh,
Neuralink is working towards the pain machine interface.
But if you ignore all that, then um, you could have,
for example, in the context of remote surgery,
you could have a specialist in a particular kind of surgery and a very advanced
robots with, um, with a very, um, um,
precise movements and precise instruments, for example.
And you could have these two things sitting hundreds of kilometers apart.
So you'll be interesting if that,
uh, could happen in my lifetime, right?
So an ability for a surgeon to perform remote surgery far away, right?
That would enable a lot, um, a lot of progress in, in medicine, in my opinion.
And I know for a fact that in order for that to happen,
you're gonna need, uh,
better computers and faster computers running on both sides.
Like just higher bandwidth communication is not enough because of a latency
constraint. You need to, um, so just,
so just to, to give you a sense, let's say it takes, uh,
20 milliseconds for any information to go from one side of the globe to another,
but humans require much less than five milliseconds feedback
time in order for you to, uh, really enhance your fine motor skills.
So that's an impossible thing, right?
Unless you had a lot of enough computing power to predict the future,
not far ahead, but a few milliseconds ahead on both sides,
in which case you could enable this sort of technology. So yes,
embedded processors, embedded systems, cyber, cyber, physical systems,
so to speak. I think, uh,
what excites me in the future sounds fascinating. Yeah. Another point of,
uh, concern, right? Is coming back to
the commoditization of, uh, computer processors, uh,
because it's a, it is becoming a resourceful commodity,
then it, this issue is actually becoming geopolitical, right?
Um, you can see the U us, uh,
passing the CHIPS act recently, which is supposed to bring a lot of,
a lot more manufacturing back to, to the United States, for example,
semiconductor manufacturing. That is. And, uh,
if you look at the supply chains of how these chips
are made, they are very, uh, let's say delicate, right?
It needs a lot of cooperation from multiple companies in multiple
different countries. And there is a lot of, uh,
different equipment that are, uh, you know,
provided by companies with a monopoly.
And so, so it's now that, uh,
it's becoming a more scarce resource, I,
I worry a little bit about what are the geopolitical consequences of that?
Are we gonna see big, uh,
big crisis as it happened during covid with, um, the microchip, uh,
shortage in, uh, in car manufacturing, for example?
So you had all the other pieces of your car ready to go,
but then because of a few chips here and there, which are unique,
and it has to come from a particular vendor,
you had to delay your whole production, for example, right?
So is there more of that in the horizon? That's something that I, I think about.
And also the, the other thing is investments.
I don't see adequate and sufficient investments yet, uh, on
non-standard, uh, or, um,
semiconductor industry for computing, right? I see. Of course,
there's a lot of investments for the semiconductor, uh, manufacturing for,
um, electronics. But for example, in photonics, it's a, it's a lot less,
of course. But I do believe that it's a technology that is going to,
uh, be part of much of this feature that I mentioned. Um, this, uh,
advanced processors, this cyber physical systems will need,
will need optical elements in it. That's something that will happen inevitably.
So I would like to see a little bit more investment on that, on that front.
Here's Anson Ho,
a physics graduate who now works at the interface of AI and other disciplines
among other deep questions.
He's interested in the efficiency of computer processing and its implications
for climate change mitigation. I.
Am an AI forecasting researcher at op. Uh,
essentially that means that I try to use tools from a bunch of different fields
like, um, economics and physics and deep learning,
and try to help us to understand what's going on in, um,
with future AI developments, how that's going to impact the world.
And we use this information to try to communicate to say,
policymakers and other AI researchers who want to ensure that the impacts of AI
are going to be beneficial. What's.
It looking like though, to you guys?
I would say I'm somewhere on the side where I'm quite concerned about like a lot
of the possible, um, misuse aspects of ai. Um,
this is originally one of the ways in which I got into it in the first place.
Um, so I'm actually quite sympathetic to like, view is that, uh,
we really need to focus on, uh, possible tail risks, um, of ai. Um,
although I don't think I go all the way to thinking that, oh, this is, um,
a situation where we're absolutely doomed and there's nothing we can do about
it. We're going to be all be dead by the end of the decade. Um, so I'd say,
um,
the most important thing for us right now is to try to gain more clarity about
these questions, and that's really what I'm hoping to do. Fair enough.
Well, how does MO'S law fit into ai? Or is it the other way.
Around? I think the framing here is to take one of, um,
thinking about AI developments in terms of a key input. So we could first ask,
what are the main things that drive AI progress?
And there are three main ingredients.
People often talk about a thing called the AI triad, um, which are, uh,
just a thing like a framework with three main components, compute,
which is a total amount of computation. It could be, um, hardware, uh,
it could be like the amount of spending on hardware. And there's also, um, data.
People often talk about big, big data, for example, there's like so much, um,
text on the Internets we can use to feed into deep neural networks. And, uh,
the third aspect is algorithms. Like what kinds of neural networks are we using?
Um,
I think right now it seems to be the case that the most important thing is
probably compute. Arguably, uh, algorithms are also quite important,
but maybe I'll get to that later. But the most important thing, uh,
seems to be compute. If you look at the trends historically, um,
we find that the between, um, say the start of, um,
machine learning or a start of, um,
artificial intelligence around 1956 with like the Dartmouth conference, um,
over until the start of deep learning around 2010,
and like 2012 was like the development of Alex Snaps, which is really, really,
really a famous paper. Um,
the trend in the amount of computation used to train these like top AI systems
has followed something similar to like a rate of progress as Moore's law,
which is around a doubling every two years.
And this seems to suggest to us that, okay,
historically we've seen a lot of progress, um, until 2012.
That is in deep learning, or sorry,
not deep learning in machine learning due to improvements in processors,
improvements in hardware, um,
just these kinds of things like slowly pushing forward. I say slowly, but two,
like doubling every two years is still pretty fast. Um,
but the reason why I say slowly is that after deep learning things really took
off After the, uh, 2012, um,
people realized that one of the ways in which you get much, much,
much better performance is really to just scale things up by a lot.
We use a lot more GPUs. We, um, invest a lot more. We also,
um, really just, uh,
have seen like a massive growth in the competition since then. So for example,
between 2012 and, uh, around 2018, we've seen, um,
a doubling time of more like six months in the training compute.
And since then we've seen a little bit of a slowing around, um,
I'd say around like 2017 or so, uh, to something more like 11 months.
But these trends just indicate to us that, uh,
there was something happening initially in the first era where we're going from,
um, 1956 to thousand 12,
mostly happening through like hardware improvements afterwards.
The big change was due to a massive increase in investments. And, uh,
we're seeing much more competition being used to train systems.
And after that we're kind of hit,
we're going to start to see some like bottlenecks where it's just hard to
continue to increase investments, but it's not, it's not really clear like, um,
how long trends are going to last and like what kinds of things are going to
come into place.
And hopefully we can try and get a better understanding of that by looking into
like the bottlenecks of different, um, that could be in place.
It could be economic bottlenecks, it could be physical bottlenecks in hardware.
So those physical bottlenecks are sort of, I mean, they're ins surpassing,
aren't they? If you can't fit more things on a chip,
then you can't fit more things on a chip.
I guess there are a number of ways around this. It is true. That is like, um,
well, with Moore's law, um, I guess we could have problems with say,
when transitions become too small, uh,
they don't operate as well because of things like quantum tunneling. Um, also,
um, yeah, there's just like problems with, um, noise,
like thermal noise that makes them very unreliable, I would say.
Like the possible ways around them are to, uh, one,
continue with the current paradigm and come up with better solutions that can
fit into the current paradigm. Um,
so you could use different types of transistors. Um, for most of, um,
I'd say the developments from 1980 or so until
2010, the transistors that were being used were MOSFETs.
So these are metal oxide semiconductor field effect transistors. Uh,
more recently we've seen like a new type of transistors that are becoming more
and more popular called Fin FETs. And these help to improve upon like these, um,
previous, um, uh, bottlenecks in, uh, microprocessors that occur due to, um,
static power dissipation, which can occur.
So this is power dissipation that is present even when we're not doing switching
operations, like doing logic operations. Um,
so this is one of the kinds of things we can do.
We can try to improve the existing paradigm, but even still, um,
there's so far you can push this.
Maybe you're saying the other other way around this is to try and come up with
alternative computing paradigms. So right now most of the stuff we're doing is,
um, using, um, irreversible operations.
And we're using things that are based on, um, you know,
semiconductors like silicon.
And we're not really thinking so much about things like, um, uh,
obstacle competing, which could also be based on like silicon,
but we're using like a different, um, type of computation. We're using lights.
Um, and this has like a,
like a range of benefits that could really drastically increase the energy
efficiency of the processors. It could also make things run faster.
And it's especially good if we're doing things like matrix multiply operations
that are, you know, really quite fixed and, um,
we can just repeat the same operation lots and lots of times. And in fact,
this is what we do for deep learning,
which just means that perhaps this is like a really,
really good paradigm that people might want suspicion into,
into the future if we want to get really, really big gains.
But so does that mean that e essentially Moore's law is
possibly gonna take off again?
Yeah, I guess there's some like differences here. Uh,
because what exactly do we mean by Moore's law? Um, I think Moore's Law, um,
people often say like frame in terms of like transistors. Um,
I think the original paper frames it in terms of this term called complexity,
uh, which could refer to, um,
components on the microprocessors that aren't necessarily transistors. Um,
I'd say from my understanding, this is not really my field, but uh,
things like obstacle transistors have really quite struggled. Um,
so maybe like the way in which we're doing the computations could be quite
different. Remember we were thinking more like a level of, um,
different kinds of circuits. So I think they're might be called like, um,
multiply accumulates. Um,
we're just using different kinds of obstacle components, um, like mock zender,
interferometers, uh, I won't get into that, but, um, essentially we're allowing,
uh, we're using different kinds of components that aren't necessarily, uh,
transistors. And so maybe there would be just something different.
I'm not sure what exactly the right analog is for Moore's Law in that scenario.
I do think that if you're looking at like a more general thing,
we're not just saying like, transistors, we could still see,
and I do expect to see something like an improvement in scaling, uh, over time.
And in fact, I think probably quite a lot of, um,
developments historically in a whole bunch of different technologies has been
just due to us like, you know, cranking, like, um, using more resources,
slowly pushing the frontiers and seeing how far we can push things until we're
able to achieve something that we want to get to. And in this case it could be,
um, building really,
really powerful artificial intelligence that we'll be able to automate a lot of
jobs. I don't know. Uh, and essentially the question then is like,
how much will we need, um, to get to these levels?
Yeah,
I wonder whether it's sort of six or eight months out from the deadline and
everyone's suddenly like, oh, quick, we better work on this so that we,
we actually hit the deadline and keep Moore's law going.
I've certainly heard, uh, stories of how, um,
Moore's law is a bit of a self-fulfilling prophecy. Um, and uh,
I'd say I'd suspect that there is like such a dynamic in place. Um,
there are pretty strong like, um, um, as far as I'm aware, there already,
already like reports by the, um, R R D S, um, that they publish every year,
just saying like, what's, what's the current states?
How can we try to continue like, making progress? And, uh,
just because it's like such a big part of like the, um, the conscious like of,
uh, people who are working in this area, um,
I think it makes quite a bit of sense that there would be such a dynamic for
people are just trying to reinforce like the existing trend. Um,
I guess there's a bit more to say to this actually, um, which is kind of, um,
that's why what people want to continue to, you know,
scale in this particular way.
Why would you want to follow that? Why would you want to make that happen?
I'm going to draw like a little bit from, um, a paper by, um, cyber hooker.
I think that's called the hardware lottery.
And essentially this points out that, um, improvements. Um,
well imagine that you are a researcher and
you're trying to come up with the next best hardware and, uh, you want to, um,
you know, earner profits perhaps. So what are you going to do? Well,
one option is to try and come up with something totally novel.
You want to be like the next, uh, genius.
You want to come up with a totally new design and just beat everybody else.
Well, unfortunately, it's quite hard to do this. Um, uh, you, first of all,
you need to be able to come up with the idea in the first place,
but even if you come up with the idea,
it's not necessarily easy for you to be able to implement it.
You're trying to beat a, a system where there's, um,
or like a supply chain where people have been, uh, optimizing,
which people have been optimizing for quite a long time. Um, there is,
there's lots and lots of money being poured into improving certain kinds of
micro microprocessors with certain architectures.
And trying to switch out of that is actually quite difficult. So in fact,
there's like a strong incentive for you to say, actually,
why don't we just continue using this particular recipe that we know already
works and we can just, you know, continue transistor scaling,
meaning we make all the components smaller. We have perhaps like come up with,
um, new techniques for, um, doing lithography at smaller scales. And, um,
you know,
if it's like a try to test a recipe and we know it's probably going to continue
for some time, why don't we continue doing it? And I'd say that's what's, um,
happening or that's probably good dynamic that's, uh,
helped sustain Moore's law historically and why we haven't like switched as
quickly to some of these alternative paradigms that I mentioned with optical and
reversible computing.
With quantum computing, is that going to take over condition, uh,
traditional computing?
Is it going to replace it in a way that Moore's law is just a thing of the past
or a quantum computer's always gonna be slightly outside of the.
Norm? Uh, quantum computing thus far has been, um,
not had that much of an influence.
And it seems difficult because of maybe three Bolton X one is just that, um,
it seems hard to come up with the right algorithms that we can apply to the
right situations and we need to find these right situations in the first place.
And it's not necessarily easy, it seems like thus far that, um,
these tend to work in, uh, certain specific domains like cryptography.
Like a second difficulty is that, um,
even if you can identify what the domain is,
it's not necessarily easy to come up with the right algorithm that allows you to
get like the final products that you have. I think for example, in um,
shores algorithm,
you have like this process of like cancellation of a lot of different things
that, um, is, that gives you like the right results, uh, at the end. And, uh,
the third aspect to this is probably just that, um,
you need to also build the hardware and make sure that the hardware is reliable.
You don't have, um, problems where, um,
like some perturbations from noise cause like the states to, um,
essentially get messed up and you're not able to do like the, uh,
quantum computations that you want to do. So my understanding is just,
it's quite difficult for us at present.
So can you tell me a bit more about the,
the sort of processing side of things and how, how that's looking? We.
Want to understand, uh, what the bottlenecks are to, um, using,
to getting really, really powerful processors and being able to use these for,
um, very exp computationally expensive tasks. So for example, um,
AI training runs, um, in order to train a deep neural network, uh,
it typically costs like a, you need like many, many GPUs for like the ta um,
AI systems of today. I think GPT four, for example,
probably costs on the order of 10 to depart of 25, uh,
floating point operations over the entire course of training, which is really,
really high, I guess. Uh, my computer certainly doesn't use, uh,
I haven't used that much. I don't, oh, I think I haven't.
How are we going to be able to continue to push, um, further if, um,
the demands for competition are just so high?
How far can we continue to push things within the current paradigm?
And these are important questions I think for, um, from two dimensions.
One dimension is that, uh,
if the energy costs associated with computation are increasing so much that they
could in, you know, increasingly have like a cause problem, say with, uh,
environmental problems or with in fact like melting the GPU that you're trying
to use, that would be quite, um, uh, problematic situation I think.
So we're worried about power density, we're worried about, um,
even if you could have really good cooling systems,
what's going to be just like an unsustainable amount of, um, computation, um,
that you would, uh, try to do but would just caused like too much harm.
And the other aspect is that let's say we want to continue improving,
what does this, uh, upper bound to, uh, harbor efficiency,
tell us about how quick progress is going to be in the future. Well,
let's say we want to determine how many, um,
closing forward operations we can perform for like a, the largest training,
ai training run using like existing processors of today. Um,
the reason why we might care about this is that, um, AI training runs depend,
or AI performance depends on these like scaling laws. So it's kind of,
if you're, if you know some statistical physics then um,
you often have heard of things like, um, scalings and,
and like phase transitions. There's like a similar thing that's inspired, uh,
by these kinds of like, um, uh,
relationships and statistical physics that people have found with deep neural
networks. Essentially, when you increase, um,
things like the total amount of training data or the size of the model or like
the total amount of training computation for these models,
we find that the performance using some metric,
which is typically like on some benchmark like the loss or like the accuracy for
classification, it tends to, it kind of scales quite smoothly.
And we can therefore use this to try to make some kind of forecast into the
future of, um, uh,
how powerful ass is going to be based on the forecast of how much data we'll
have and how much computation we'll have. If we then say, well,
I might expect that there's going to be like a really,
really large amount of computation,
say like tentative hour 40 focusing operations,
let's just say that we believe this for the sake of arguments,
is it going to be feasible for us to do this? Um,
turns out it's, uh, going to be quite difficult, uh, given existing, um,
microprocessors. Um, so, uh, project that, um, some, uh,
me and my colleagues have been working on recently, uh,
we roughly estimate that we're probably going to hit some bounds on the order
of, uh,
performing 10 to the power of 16 floating point operations per juul of energy
dissipated, which is about like three orders of magnitude. Uh,
I'd say three or four orders of magnitude higher, uh, than existing, uh,
microprocessors. And this just means that, well, if we're going at, uh,
current rates of progress,
I believe we're doubling on the order of like two to three years for energy
efficiency like floating front operations per juul. Uh,
we're going to hit these limits like pretty quickly. Um,
or before we hit these limits, progress is going to have to slow.
And if this is the case, it just means that, uh,
if we want to do this like tentative power 40, uh, flop training run, uh,
we're going to, uh,
if you look at the total amount of jus of energy that are released, um,
really it just means that we're probably going to, uh,
emits on the order of like the same amount of energy that's emitted from the
earth every single year, which I believe is on the order of tentative,
however of 24 Jules. Um,
and that's about four orders of magnitude higher than, uh, the amount of, uh,
energy that's released through like fossil fuels every year, say think in 2021.
Wow. Yeah. Uh, so this just means, well,
there are some well restrictions on what we can do from these energy efficiency
limits, but of course you might ask, well,
maybe we don't even need to get to these like tentative power 40 like, uh,
flop training runs. These are, that's just a really, really large number.
And in fact, I tend to agree, uh,
the reason why you might believe that tentative power 14 might be relevant in
the first place is that you might think, well,
let me just try to guess how much computation you'll need to build like a
really, really powerful AI system by looking at evolutionary history.
If you look at evolutionary history and you count like all of the floating front
operations and quotation marks, , uh,
they were like done in the neurons of organisms throughout this period. Um,
we find that it just take, it takes like a really, really large amount.
It's on the order of like 10 to the 40 something, uh, floating point operations.
And so this just suggests that, um,
if it is the case that we should be anchoring our estimates of the compute
requirements from evolution, um,
that we're going to have some problems with heating if we're using existing
microprocessors. And perhaps this could mean that there's going to be like a,
in the future,
a fairly strong incentive to switch towards alternative paradigms,
which are orders of magnitude more efficient. And on the economic side,
and first I'm going to like try to explain this in the context of, um,
my previous economic paper, which is quite famous. It's called, um,
our Ideas is Getting Harder to Find.
The idea here is that as you make more and more progress, um, in like, uh,
doing some kind of r and d in some domain, um, because most of like the,
uh, initial earlier ideas are getting fished out, uh, you know,
you're picking all the lowest hanging fruits, um,
further progress becomes harder and harder.
You have to go and like climb higher in the tree, I suppose, uh,
to get like the fruits. So how large is this effect is the question,
does this mean that like, uh, researchers are becoming less and less, uh,
productive,
are able to come up with fewer ideas and make less progress over time? Um,
in fact they find, uh, the paper finds that, um, this is the case, uh,
for quite a wide range of, uh, fields. It's true across the aggregate economy,
uh, in some like frontier economies that they look at.
It's also true within individual sectors, um, including, um,
I think semiconductor r and d. Uh,
they also look at things like Moore's Law in their paper.
And why is this relevant? Well,
it tells us that even without thinking about the energy efficiency limits,
we get these like kinds of like diminishing returns to, uh,
as we get doing more and more research,
it becomes harder to make further progress.
Now when we bring in the energy efficiency limits, it becomes much, much,
much harder as we get really, really close.
The ideas are just going to dry up like much quicker if we're assuming that
we're continuing within the same paradigm.
And this means that we're going to have to massively increase the amounts of
research inputs in order to be able to make further progress at say the same
rate. It's becomes,
Moore's law in a sense becomes much less sustainable and it becomes harder to,
um,
do that unless we're able to have a lot more researchers that are able to come
up with new ideas. And typically, I think historically, um,
some of the best economic theories that we have tell us that, um,
a big reason for this is like an increase in population or the increase in
number of scientists which have been able to like compensate, uh,
for these difficulties. Although I will say that this is a little disused. Um,
yeah, so that's the second way, uh, in which, uh, we have um,
some kind of an influence here. And the way we could, one way we could try to,
uh, get around these bottlenecks is to bring artificial intelligence into the
picture although human populations, well, I'm not sure how, um,
much we're going to be able to, um,
increase like human population at the same rates as say, like in, uh, the 1960s,
uh, especially when, say like fertility rates. And, um, some of the, uh,
frontier economies right now are, are, say, in Spain,
I think it's down to like 1.4 in South Korea, it's around 1.4 as well. Um,
it just means that, um, we're, it's going to be harder to get enough humans to,
um,
have like these ideas to do research and to continue pushing things at the same
rates. But with artificial intelligence in the picture, maybe AI can help us,
um,
do like research and come up with new ideas or to augment the productivity of
these researchers such that we can sustain, um, uh, meets this like increase,
uh,
for some time and push things a little bit further until we get to the limits.
And then maybe then we will still have to switch to alternative paradigms. Yeah.
Okay. Can artificial intelligence come up with ideas?
I tend to think that it can, like, I don't, um, I would say like,
it's not entirely clear, like, uh,
there are cer there's certainly some uncertainty. Uh,
and I think we should be thinking about this using some kind of like a
probability distribution.
Like we have some guesses and perhaps like in the using some kind of, um,
Bayesian statistics approach, um, we have some, uh,
initial guess like a prior distribution. And then based on like our evidence,
uh, that we might have, like we might see that's, you know,
GPT four was actually surprisingly good at some cost that I didn't expect it to
be good at.
And then I might update my beliefs towards the direction of thinking that
actually it could be the case that it's, uh, pretty intelligent. Like it can,
it has, it's able to do some kinds of common sense reasoning. Um, for example,
I think, um, uh, , uh, like, um, one of the famous like, uh,
founders of like deep learning, uh, made this claim, um, that, uh,
if you ask g PT four that,
or if you ask like a really powerful language model that is something like G P T
four, if you put like a book on the table and you push the table,
will the book move along with the table? Um, uh, he predicted that, uh,
GVT four wouldn't be able to guess that the book might move along with the
table. Uh, but I think this is not true. Actually, if you just asked gvt four,
uh, it actually some has like this kind of like, um, you know,
some kind of a picture. I don't know if it's like an actual picture,
but it's able to answer this kind of question with some degree of accuracy.
So I think these kinds of things,
like if we actually give it a test and we see that it's able to do, um,
solve these kinds of problems,
I think it will be able to do similar kinds of things. Um, and in fact,
I think I would guess personally that if you're able to scale to really,
really large scales with massive networks like on 10 to the 50 parameters,
which is way larger than anything we have today, uh,
by many orders of magnitude, um, I think we could do that. The question is,
well, in practice we'll be able to do it.
And I would guess that there are probably more efficient ways of, um,
training our artificial intelligence than what we are currently doing today. Uh,
and you know, I think in practice, if you think, uh,
from like an algorithmic perspective, um, even really, really, um,
I guess like not particularly intelligent algorithms are able to perform really
complicated tasks just by using a large amount of computation to compensate for
it. So I think there's probably something similar here, but I, of course,
I can't be certain. Yeah.
Who would you like to know about all these issues
that are forthcoming?
I guess I'd be most interested in like, um, people who are, uh,
maybe were in like a similar position to me, I guess. Uh, so I only recently,
uh, got into, um, AI forecasting and thinking about the future of ai. Uh,
I actually finished my undergrad at, um, university of St. Andrews, um,
back in 2021. Um, and I was just wondering what am, oh my God,
what am I gonna do with my life?
But I also wanted to really just like figure out what, um,
things I could do that were impactful and were like big questions to me, uh,
I think could have like an influence on like policy, like it affects like, um,
the rates of, um, progress in from like an economic perspective. And, um,
it has, um, influences for, um,
people who want to decide like how to govern AI such that if the progress is too
fast, like there's going to be problems.
So we want to make sure that we can adapt to these changes.
Like these are questions that help inform us about that. And I really feel like,
um, that's, you know, there are some ways in which we can try to use like, um,
things like the skill sets of physicists and solve really,
really important problems. I mean,
I think it's a bit presumptuous of me to say like, I'm doing, like solving, uh,
important problems. Like I, I try to do, so I don't know if I actually am,
but I think more people should be thinking along the lines that they actually
can do it. Um, I think a lot of my friends at, uh, uni and myself included,
I would say actually with, um, that we were initially thinking that, well,
what can we even do? It seems really hard. The only thing we can do is, um,
to maybe think a little about about climate change, but, um,
and that's important . Uh, but, uh,
are there other things that we can do? There's so many problems out there that,
uh, could benefit so much from people with, um,
who are mathematically minded or, uh,
have some kinds of intuitions from physics.
There are so many problems out there. Why don't we try to, um, do this more,
I guess this would be the, um,
like people who are interested in trying to figure out how to impact the world
in a positive way and to really use, um,
maybe physics or maybe to switch to another thing, to,
I'd like to talk to people who are, uh,
to really communicate with people who are, um, interested in this kind of thing.
Louis Barson has a message for you, the listeners.
I mean, I think, you know, if they're physicists listening, uh,
and they don't know about semiconductor industry, please do, you know,
read up on it. I think it's a,
a fantastic example of where physics is right at the heart of, uh, you know,
economic and societal transformation. Uh, and it's a fantastic story, um,
even if it isn't the area you wanna go and work in, uh,
although of course it's a great area to work as well. Uh, so please, please do,
uh, read up and if you haven't, uh, already, uh, heard the story, and, uh,
if you do already know about it, um, then, uh, please do participate in, uh,
the, the IOP's, uh, impact projects because, uh, we really want your, uh,
your voices and your views to be, be reflected in, uh, you know,
the areas where the i p is trying to make a difference on the,
on the national stage.
If you'd like to know more about Moores Law,
you can find an article on physics world.com by James Mackenzie,
and we'll be back next month when we'll be looking into the International year
of basic Sciences for sustainable development.
And thank you very much for listening.
Physics.
World.