-
- Physics world.
- Hello and welcome to the Physics World Weekly Podcast.
I'm Hamish Johnston.
Artificial intelligence seems to be everywhere these days,
and science is no exception.
But how can physicists best use the rapidly
changing technology, and
how can they be confident in the results AI delivers?
To find out physics world's, Catherine Skipper spoke
to the co-founder of an international consortium
that's developing AI tools for scientists.
- In 2022, KO's Deep Mind Group announced
that the Alpha Folds artificial intelligence program had
cracked one of the thst problems in biology.
How to predict the structure
of a protein from the amino acid chain.
AI is already changing the way we do science.
It could open up new possibilities in everything from
particle physics to drug discovery.
But what's the best way to use AI in research?
How should scientists share knowledge and resources,
and what responsibilities do scientists have when it using
artificial intelligence To talk about this, I'm joined
by Rick Stevens, who is the Associate Laboratory Director
for Computing Environment
and Life Sciences at the Argonne Laboratory in the us.
He's also a professor of computer science
at the University of Chicago.
Rick is one of the founders
of the Trillion Parameters Parameter Consortium,
launched in November, 2023.
It's an initiative to bring together researchers from all
over the world to coordinate the development
of AI tools for science.
Those involved include scientists at the SDFC,
Ruthford Appleton Laboratory in the uk,
and the founder span academia industry
and research institutes.
Rick, thank you for joining me.
- Thanks. I'm glad to be here.
- Uh, so firstly, the number 1 trillion is something
that I see a lot, um, and people talk about ai.
Can you explain what the significance of that number is
and what trillion parameters means?
- Well, it's a big number, means a million
million 10 of the 12th.
The reason it comes up in AI research is it's, uh, a number
that kind of characterizes the scale of current, uh,
frontier level AI systems.
So you hear about systems like chat, GPT-4 or Cloud three,
or Gemini Ultra and so on.
These models are, uh, expected.
I mean, a lot of the details are not public,
but, uh, from what we know,
these models are in the trillion parameter, uh, regime,
meaning that they have, uh, 10 of the 12th, uh, weights
or variables that were trained, adjusted during training
to represent the knowledge that the model contains,
the model, the knowledge,
and the skills that the model contains.
So that's a scale that's quite interesting.
Very commonly available models, uh, open models tend to be
more in the 10 billion parameter range.
So maybe a factor of a hundred smaller than these, uh,
state-of-the-art models and a a trillion parameters is a,
is a very challenging model to actually create and to train.
So for example, uh, if you wanted
to train a large language model with a trillion parameters,
um, it's likely to take on the order of many months on, uh,
a machine that can do over, uh, an XA flop worth
of computing, uh, per second.
Um, so maybe two or three, four months.
Um, and so it's a very large computation.
Um, so it kind of represents the frontier
of accessible ai.
Um, most academic groups, I would say,
do not have the ability to create models that large,
they don't have enough computing resources to train them
and to train a model that has on the order of say,
a trillion parameters, you would, uh, want
to have something like 20 trillion tokens of data.
So that is a huge amount of data.
Uh, we can talk about what that means.
But, uh, so these are large, the very largest models
and the very largest collections of data to train them.
- Right. Um, so now onto the, the consortium.
How, how and why was it founded
and what are the, what are the aims of the, of the project?
- So, in the last year
or so, um, there's been, uh,
an increasing interest in the scientific community
to think about building, uh, large AI models,
generative AI models, whether they're language models for,
uh, scientific text,
or whether they're models
that might operate in biological sequence space
or weather, uh, field space or something like that.
Uh, to build such models, uh, for scientific purposes
and to push the scale
because it, what we know when you're building AI systems
that scale matters.
Scale is one of the magic, uh, elements that makes ai,
uh, happen.
Um, we can build small models,
and when we scale them up,
they behave qualitatively different than
differently than the small models.
And so scale is a, is a big factor, uh,
in imagining models for scientific purposes
that would have considerable more capabilities than the
current models that we have.
Anyway, there's a lot of people thinking about this.
And, uh, unlike the, the handful of very large companies
that have the resources, uh,
to train such models internally, uh, a lot of, uh,
of our colleagues, uh, are,
we're interested in working together on arranging, uh,
arranging collaborations in a way
that we could either work together on creating data sets
to train such models work on sharing knowledge around how
to build such models and, uh,
to some degree work on sharing access to computers
that now the research community does have access
to large computers, like, like the exascale system
that we're building at Argonne,
and similar systems, say at Oak Ridge
and other laboratories around the world,
like our colleagues in Japan and so on.
Um, but these machines are heavily, uh, subscribed
and for the academic community to, to get, say,
a large allocation of time on a academic kind
of supercomputer to use for training ai,
there would probably need to be, uh, many groups agreeing
to work together on that in order
to make it worthwhile to allocate that time.
So the Trillion Parameter Consortium is really, uh,
an organization to try
to cross-fertilize all of these issues.
You know, how do we build the data sets?
How do we architect the models?
How do we work together on acquiring access?
How do we evaluate the models when we build them?
Uh, how do we ensure that they're safe, uh,
and how do we even understand, um, uh, the avenues
of progress in the, in this, uh, space. Right.
- And so, talking a bit more specifically about, I guess,
about about physics, um, yeah.
And you spoke about, you know, how much of a difference the,
the scale of that you're able to access makes.
What kinds of problems in physics are people thinking about
when it comes to ai?
And yeah, how much
of a difference does it make having access to say,
a trillion parameter model versus the computing power
for a couple of billion parameters?
- Well, one of the key ideas in generative AI
for science is this concept of foundation models.
So a model that's trained broadly on data from, uh,
many problems, many sources, many domains, um, in physics,
there's an effort, uh, to think about this
or to build a model along these lines
for high energy physics that imagines training a single
foundation model on hundreds of experiments worth of data
or many different instruments, many different accelerator
and detector systems.
And then, uh, use that model for many downstream tasks,
for identifying particles,
for identifying trajectories and so forth.
So this idea of a single kind of, uh, supermodel
that is, uh, trained very broadly
and then could be adapted for, uh, dozens
of downstream tasks is, is this concept of,
of like a foundation model.
And the different disciplines are making plans
to build foundation models for their domain.
So you could imagine a model like
that being built in, say, high energy physics.
You could also imagine a model like
that maybe being built in cosmology
or in astrophysics, um, for to be useful in a range
of problems in, in that domain.
Uh, we know of people building them
for materials science, for example.
Uh, same idea. So a single model that's trained on, uh,
thousands, hundreds of thousands, millions of, of, uh,
say material samples, uh, crystal structures, uh, you know,
physical measurements, crystallography data, and so forth.
And then that model would be able
to make predictions in many different,
uh, targeted application areas.
So that's the concept. Um, so that's
what we call the concept of these domain foundation models.
So that's, that's one idea
that the trillion parameter consortium members
are, are working on.
There's other approaches where you want
to integrate data from many domains into a single model.
So think of it, uh, like you're trying to build a, a model
that's kind of a polymath, right?
That knows, uh, deeply it knows applied mathematics
and knows physics, it knows biology, it knows chemistry,
it knows material science, it knows climate and so on.
And the idea of that kind of a model that goes
beyond just say, a single kind of physics domain is, is
to be able to integrate
and synthesize knowledge in a multidisciplinary fashion.
So to bring patterns
and structures that might occur in physics to biology
or from, uh, biology into, uh, some other, uh,
you know, from say, biomedicine into, say,
synthetic biology or something like that.
Um, so this idea of models that are interdisciplinary
is also very powerful.
And of course, the large, uh, general purpose
language models that we have access to public, you know,
publicly accessible models tend to be in
that second category.
They're broadly trained on, on many domains.
So Chad g PT four is, is a, you know, pretty good in physics
and it's pretty good in biology,
and it's pretty good in chemistry.
Um, and so it's an example of a model where
you could expect it, uh, maybe future versions to be able
to integrate concepts across domains.
So I see two different kind of directions,
and then we see kind of a third idea emerging
where you mix these concepts.
So maybe a, a model that's very general purpose
that's interdisciplinary.
It knows the language of many parts of science coupled
to a collection of domain specific foundation models.
So it knows, say that the hydrogen physics model knows how
to do certain things, and the material science
model knows how to do certain things.
And depending on what you're asking, the, the system to do,
it will vector those requests to the individuals
and expert components.
So all of that we think is possible.
The consortium itself isn't really trying
to advocate any particular way of doing this.
It's really just trying to bring the community together
that have common interests and promote best practices,
and also, uh, raise awareness that there's huge opportunity
for AI in science.
And, uh, it's gonna represent, uh, a direction
that will be incredibly fruitful over the next 10, 15 years.
But it's also gonna be very resource intensive.
And, uh, part of the consortium's goal is
to help educate funding agencies
and governments around the opportunity
and the need to invest in at scale.
- I, I guess, as well as the fact that AI can be used
to solve specific problems.
When I, when I think about, when I think about the use
of AI in science, I think about, you know, I've just,
so I've just done a PhD and I think about
how I feel about the thought
of doing research without the internet,
which is not a long time ago at all,
but it feels unimaginable to have, you know,
to have done research where you have
to print out every paper that you use
and where you don't have access, instant access
to all your data, and you can't video call
with collaborators
and you can't use Google to ask questions.
So, as well as like specific, specific uses
of artificial intelligence, how do you think
AI will change just the day-to-day way that, um,
that research has done?
- Well, that's a good question
because we we're already starting to see it happen.
Um, where instead of using Google, say to find a paper,
uh, depending on the AI you're using, you might be able
to ask it to build you a bibliography on a topic,
and it can go out and find the papers,
it can read and summarize the papers.
It can select the papers that are most relevant
to your question and build you a bibliography.
And by the way, download the papers for you
if you want to go deeper.
Um, so this idea of using AI to accelerate
what we already do is one way to use it, another way
to use it that, that we're working on,
and some of our own work is having a system read, say,
50,000 papers and extract information from those papers
and integrate that information to extract,
say hypotheses from tens of thousands of papers,
analyze those hypotheses and try to find what's common
and what's different between them.
Try to identify, say, experiments
that might answer more than one hypothesis
or address the questions in one or more hypothesis.
That's something that would be almost
impossible to do without ai.
I mean, it's individuals, you know, in your lifetime
you might read 20, 30,000 papers.
Uh, but the idea that in a weekend a system could analyze,
say, 10,000 papers and,
and say, here's the 30 hypothesis
that are most important in this field.
By the way, here's the 20 experiments
that would inform, you know, progress.
You could do that in a weekend.
Now of course, you have to do the experiments maybe.
Um, but this kind of, uh, acceleration
and integration across all of knowledge becomes possible.
Um, so in, in the literature,
I think this has becomes quite doable.
Now, a lot of scientific data though, is not
yet available to ai.
So one of the challenges that we have is there's exabytes
and, uh, petabytes and,
and exabytes of scientific data
that are not in papers, right?
The raw data from experiments
or the biological databases that underpin, you know,
genomics and so forth.
And one of the challenges
that we're addressing in the train parameter consortium is
how do we expose that data in a way that
AI can have access to it?
Um, so we see lots
of different ways in which AI will
accelerate and support science.
I mean, one way to think about it is, imagine the, a little
AI assistant that helps you that you have access
to 24 7.
And it can run experiments, it can read literature,
it can summarize, it can maybe talk
to your colleagues ais about what,
you know, what they're up to.
And, and every morning gives you a little
progress report on, you know, I talked to these ais,
you know, your colleagues, and, um,
and here's the new idea to think about today, or,
or, uh,
we made progress on these problems you gave me from last
week that we think is possible.
Uh, it'll require a lot more computing and lot
and more powerful AI capabilities than we have today,
but we see inklings of that emerging.
- Right. And actually, something that I also thought about
as you, as you were talking about using data
and the sheer volume of data that we have something
that's been a topic of discussion in physics and,
but I know in other areas of science is the issue
of reproducibility,
and I guess how, how good the, the massive volume
of data we have is, do you think that's something
that's gonna get more important as we're training AI models?
- Uh, well, I think it's important to train models on data
that are self-consistent and correct.
A lot of data, particularly in, in fields that are
benchtop level science.
There's a lot of variability in the quality of data.
There's two ways to think about this.
One is how AI might be able
to sort out the good data from the bad data
and try to make some help.
You make choices in terms of what data
to consider in your, in your questions.
But another direction is for the AI actually
to run experiments, right?
So we're quite interested in this idea
of autonomous discovery,
or what some people call self-driving labs,
where the AI is actually taking the problem
from the scientists, uh, typically
and translating that into, uh, protocols
and then translating that into, say, operations
for the robots and carrying out the experiments, uh,
aggregating the resulting data, analyzing the data,
maybe suggesting more experiments and so on.
But the more we automate, say, laboratory level science,
the more repeatable, the more reproducible
that science will be.
Uh, 'cause we're kind
of factoring out the, the human element.
Now, there still could be other reasons for variability,
you know, weather, air pressure, you know, just, uh,
humidity, you know, things that kind of vary, uh, from day
to day that that also inject noise.
But this idea of moving from humans, being the,
the mechanism for doing experiments to say automation,
robotics, and humans may be more focusing on the broad set
of questions and the relationships that you're trying
to establish could improve reproducibility.
But a lot of parts of science, there's still a lot of, uh,
uncertainty in how we do things, right?
So in biology in particular, experiments can be challenging,
particularly quantitative experiments,
and AI may be able to help a little bit with that.
But, uh, as our, you know,
what really holds us back is the fact
that we still have a large amount of, of lack
of understanding in certain systems.
And so these systems might be noisy
and they might be very complex,
and so, so we have to attack all of it,
but I think AI will help on almost all those
dimensions. Yeah,
- That's, that's interesting.
I hadn't thought that you might be able to, I guess,
automate a lot of the, a lot of the stuff
that PhD students are sort of doing now that might not be,
might not be the most, the most sort of fun
or intellectually stimulating parts of their, of their work.
I remember I spent a lot of like long evenings,
like in the basement of the lab, like running experiments.
It would've been quite nice if I could have
gotten a robot to do that.
Um, yeah, so we've, um, we've talked about, again,
the potential of, of artificial intelligence to solve,
you know, problems in science.
But I think it's another, another side of that is
that it's an incredibly powerful tool
and it, you know, an incredibly energy intensive tool,
and it comes with, you know, responsibility.
So, so what kind
of responsibilities do you think researchers have when using
and developing ai,
and how is the consortium working on that?
- So there's, yeah, I'd say there's a
couple key responsibilities.
One is that, um, as a, as scientists as, uh, creators
of new technology, we want to make sure
that we don't create things that are intrinsically dangerous
or will be used for bad purposes.
So this idea of AI safety is very important.
Um, if you're building models that deeply understand, say,
molecular biology or infectious disease, they could be used
to create cures for diseases,
but they also could be used perhaps
for engineering, uh, pathogens.
And so how to, um,
safeguard against the bad use cases,
I think is a major challenge.
And I think we have some notions of how to do that,
but we're, we still need a lot more
research and how to do that.
Another one is this idea of, you know,
in science we need things to be reliable,
truthful, to be repeatable, of course.
Um, and that means that we ideally want ais
that are grounded in the world,
that have put a special value on
data from the real world as opposed
to from some other source, right?
Um, and I think how, how we do that
is still pretty open, right?
So there's some approaches where you try
to build the physics into the AI itself so
that it can't violate the laws of physics
or it's penalized for violating the laws of physics.
But I'm more interested in the case
where the model understands physics
and you can negotiate with the model what physical laws
or what physical principles you want it to,
to pay attention to.
And if you have a model like that, you can use it
to explore alternative theories.
You can use it to explore alternative physics.
Um, and I think we, we need to move in that direction,
not just put guardrails on the model to say, uh, you know,
be straight locked, uh, straight jacketed into, uh,
some current physical notion.
Um, but
to have the model actually understand when it's following
known laws of physics and when it's outside of that, uh,
space would be super useful.
Uh, we want models to be tools in theory development, right?
And, and to be able to compare alternative theories.
And that requires much more, uh,
capability than we have in, in current models.
Um, we need them to be really good, uh, at,
at doing detailed calculations as well.
And a lot of, uh, current language models are,
they're getting better, but, you know,
they can still make very simple math errors.
Um, it requires a fair amount of work to get them
to be, uh, rigorous.
Um, and that's, it's improving
because there's ways of having the model talk to, say,
a symbolic computing engine or a calculator
or a, you know, write a computer program to answer something
where maybe the numerics are better calculated directly
than, than say from the model, uh, trying to, uh,
compute a completion or something on that.
So I, I mean, there's, there's a special set of needs,
I think, for science that distinguish, uh,
AI properties from the general ones.
And in the TPC,
there's working groups discussing these topics and,
and working together on them.
- Right. Thank you. So just finally, can you finish up
by talking a bit about
what's the trillion parameter consortium is gonna be working
on in the near future, in the next year or so?
- Well, there's some workshops that we have planned, uh,
in Europe this summer, um, to bring people together
that we're still in this stage of,
and tri, we just came out of some meetings in Sydney, uh,
a couple weeks ago that were associated
with the Supercomputing Asia conference, where we had,
you know, a hundred people in, uh, meetings, uh, tutorials
and planning sessions around data, uh, aggregation,
around model development, around evaluation,
these things I mentioned.
So we'll have some meetings in Europe this summer, um,
that will be announced.
Um, we are, uh, currently having combinations of
tutorials on generative AI for different scientific topics.
So that's usually part of meetings.
And then working group sessions.
A number of, uh, TPC members are
actively training large, uh, models.
So the Allen Institute for AI recently did some work,
a model called, uh, oma.
Um, we've got a model that we're building, Aurora GPT, uh,
my lab, uh, where there's other models being built
by other members and, uh, sharing results
and sharing data associated with those models.
So it's kind of, all of those things are likely
to continue for the next year.
- Right. Thank you for coming on the podcast, Rick,
and talking about your work.
- That was Rick Stevens of Argonne National Lab
and the University of Chicago.
In conversation with Katherine Skipper last year, I spoke
with the theoretical physicist Matt Hodgson, about
how he was using artificial intelligence in his research
and teaching at the UK's University of York.
You can find that conversation in the archives
of the weekly podcast.
Just look for the headline, the Chatbot Revolution,
how physicists are using Large
Language Models in Academia.
I'm afraid that's all the time we have
for this week's podcast.
Thanks to Rick and Catherine for a fascinating discussion.
And thanks to our producer Callum Gel.
We'll be back again next week with an episode
that's dedicated to ATO second Science.
But in the meantime, do check out the latest episode
of the Physics World Stories podcast
host Andrew Gluster explores the science, politics,
and ethics of the Netflix Series Three Body Problem
with the show's Science advisor, a linguist
and a science writer as his guests.
You can find that episode on the Physics World website
or at your favorite podcast provider.
Just look for the headline Three Body Problem, a deep Dive
into the Netflix show,
- Physics - World.