Hello, and welcome to the Physics World weekly
podcast. I'm Hamish Johnston.
It's the September 18, and we're in the
middle of peer review week.
And this year's theme is rethinking
peer review in the AI era.
My guest in this episode is Laura Fiethom
Walker, who is reviewer
engagement manager at IOP Publishing.
As well as publishing Physics World,
IOP
produces over 100
scholarly journals,
and it's just reported the results of a
new worldwide
survey of reviewers' attitudes
to the use of artificial intelligence
in the peer review process.
That report is called AI and peer review
2025,
and Laura is the lead author.
Here's our conversation.
Hi, Laura. Welcome to the podcast.
Thanks, Hamish. It's great to be here.
So, Laura, before we talk about, this survey,
can you just give us,
an idea of what IOP Publishing's
current policy is on the use of artificial
intelligence
in the peer review process?
And is is is the policy in line
with most other scholarly publishers?
Yeah. So it's a good question. Our current
policy
is prohibitive. So we
do not accept or condone the use of,
large language models to
write peer review reports at all or to
edit peer review reports.
And
that's because we have a lot of concerns,
ethical concerns about uploading confidential manuscripts into these
AI chatbots. We don't know,
where that information is being used.
It's also because, as we'll talk about in
a moment,
views of the scientific community are actually really
polarized,
and there's a large proportion
of researchers
who do not feel comfortable
with AI being used to review their manuscripts
in any way.
And so we went with,
a prohibitive
policy.
There are lots of publishers who have the
same policy,
but, actually, when you look across the industry,
there's a an enormous amount of variation
in what different publishers are mandating.
Another common approach is to say that,
some publishers accept the use of,
generative AI to kind of do light editing,
language editing, grammar editing, things like that, but
they emphasize that the reviewer is
ultimately responsible for the content of that review.
Obviously, it's very difficult to police that. It's
difficult to police
the extent to which,
a reviewer might have used a large language
model.
And there's another another
large proportion of,
publishers whose policy is that
reviewers can use
large language models
to write or edit their reviews, but they
have to,
be honest with
the
publisher and with the authors about how they
were used. So they have to
usually tick a box or make a statement
to say to say how they were used.
I think
what we really need as an industry is
some kind of harmonization of these policies
because, you know, often reviewers aren't aware of
the specific policy of the publisher that they're
reviewing for. And,
there are so many publishers, they might be
getting lots of lot lots and lots of
different requests for different journals.
I think we need to work together a
bit more so that we're all on the
same page, and that will help reviewers, and
it will also help authors.
And, I mean, it sounds to me that
surveys are really needed, aren't they? Because,
you know, as you said,
reviewers
are polarized.
I'm guessing that authors are polarized as well
in terms of whether they want their,
papers to be reviewed,
using AI.
So
with this survey that, that you've just concluded,
who did you speak to?
How many respondents
did you have, and what kind of questions
did you ask?
Sure. So we put this out to our
peer review community. So that's people who have,
been invited to review for us or reviewed
for us in the past.
And we got about 350
responses.
It was a really diverse group of respondents,
really geographically diverse, a good mix of gender,
a good mix
of, career levels as well and subject areas.
So we were quite pleased with the diversity
of the responses we got.
And
it was quite recently that we did another
survey. Essentially, in 2024,
we did the state of peer review report,
which asked
much broader questions of the peer review community,
and one of the questions we asked was
about the use of AI.
And this is such a fast moving field
that we really felt the need to go
out and ask more detailed question
questions about this a year later. And what
we have found is that things have shifted
even in that quite short space of time.
And I think we, as publishers, need to
understand what our communities are feeling
and to kind of
stick with them throughout this process and adapt
our policies accordingly.
So,
so we we asked a whole range of
questions of these 350
people.
One of the the main insights was we
asked them the same question we'd asked them
in 2024,
which was,
what do you think the impact of generative
AI will be on peer review?
And we did see a shift. So far
fewer people in 2025
were neutral.
In the 2024
survey, 36%
of people said, well, I don't think it's
gonna have much of an impact.
That had reduced to 22%
in 2025.
So a 14%
reduction, and
those,
respondents had kind of gone to the either
end of the spectrum.
So there was a 2% increase in respondents
who thought that,
AI would have a negative impact overall, but
a 12% increase in respondents who said that
they thought AI would have a positive impact
overall.
And it's important to bear in mind this
is a different,
sample. It's a different population of people, and
it's a smaller sample size. So it might
just be that we we kind of got
a different mix of people.
But I do have a feeling that
things are becoming more polarized, and it might
be a a case of more people are
aware
of these tools now in the last year.
And,
you know, they've thought a bit more carefully
about the impact that it might have on
peer review.
I see. And, I mean, I suppose,
it it does make sense, really,
that this polarization
will increase
maybe simply because people are becoming more aware,
as you say,
you know, because I I suppose AI has
become normalized
almost in in everyday life,
for a lot of people. Do do do
you think it's it's that and maybe a
combination of of the fact that people are
be have used AI, and so they know
what it's capable of doing and that they
sort of project that on
on how they could use it in a
review or how they wouldn't want it to
be used in a review.
It's a really good question. The question of
to what extent are the research community
truly aware of
what these large language models are capable of
and what is a maybe a good use
for them in the peer review process and
where they shouldn't be used. We asked,
lots of free text questions, and we analyzed
those responses, which were really interesting.
And, again, they were very polarized. There were
a lot of people who raised serious ethical
concerns.
But there were a lot of people who
said,
well, you know, I use large language models
all the time, and, actually, I use them
for analysis, and I use them to
analyze the manuscript under review.
And, really, when you understand how LLMs work
kind of under the hood,
they really shouldn't be used for scientific analysis
and critique. It's not something that they're capable
of.
What large language models do is they predict
what is the most likely next word, and
they're capable of producing text which is very
convincing
but not necessarily very accurate. And that's something
we see a lot when we look at
fully
LLM produced
peer reviews.
So
I I'm not sure that the re the
physical science research community, certainly the,
some of the respondents to our survey,
I'm not sure they are fully aware of
how these tools work and what their drawbacks
are, you know, where their blind spots are.
Yeah. Well, that I mean, that sounds like
a problem, but, I mean, I'm guessing, you
know, you're you're dealing with, you know, with
some, I suppose, smart tech savvy people. So
at some point, I I the community will
probably have a a better
realization,
of the issues involved and what AI can
and can't do. Mhmm. Another interesting thing about
the survey is that it it reveals a
split in the views of early and later
career
researchers.
And, you know, perhaps not surprisingly,
early career researchers tend to be more positive
about the impact
of AI
than their senior
colleagues,
whereas later career respondents tend to be more
neutral
about the possible
impacts.
I I mean, is that
I mean, I know this is sort of
stereotypical
that, you know, maybe the early career
researchers are are more savvy when it becomes
to AI and and actually use it, where
maybe
people further on in their career aren't
picking up
new tools, you know, they're happy with the
the way that they've done things in the
past. Is is that the reason why, or
is is that just a horrible stereotype that
I've,
that I've un unveiled?
I think it's a very good question. I
mean, we did this sub analysis, and, actually,
we analyzed the question
about what do you think,
the impact will be. So that's where we
looked at the the generational divide.
And, yeah, you're completely right. So early career
researchers were much more likely to have a
positive view of the impact the future impact
of generative AI.
And it may well be because they're digital
natives. They're generally just more comfortable with these
kinds of online tools.
But I think it's also
important to consider that early career researchers now
are starting their careers in a very different
landscape.
You know, academia,
those early career researcher jobs look very different
to how they did
thirty, twenty, even ten
years ago. The volume of,
administrative
tasks is is is much higher that lots
more is expected of their time.
There's a lot more precarity
in their work, and and so
it might
be a product of the fact that early
care researchers
are less likely to have experienced
peer review as it traditionally should be, which
is a positive experience where,
the authors receive really useful and supportive comments,
which help them to improve their manuscript, which
help them to improve as a researcher.
And,
you know, and then they can go on
and get their paper published. And and,
this is purely anecdotal. This is just my
opinion, but I wonder if that's also a
factor. They haven't experienced
peer review in the same way that their
senior colleagues have.
I I think there are some very interesting
dynamics
in,
you know, how people,
adopt AI and why they adopt it. So,
yeah, I suppose more more surveys are needed
there as well.
Absolutely. I mean, we you could just do
you could ask so many questions. I think
this is so fascinating, and things are moving
so quickly.
What I really enjoyed
reading
was the kind of, free text comments,
because
it was very clear looking at those. In
fact, I can I can read you some
of them,
here?
Oh, go ahead.
So we asked
respondents
to tell us whether they thought there were
any ethical issues around the use of generative
AI in peer review, and
there was a real range of responses. So
I'm just gonna read you a couple.
The theft of the corpus of data
used to train AI models, the replacement
of human labor, and the wasteful energy usage.
That's one comment.
Another comment was
the main ethical issue is the transfer of
responsibility over knowledge from a human intelligence to
a nonbiological
intelligence with unknown administrators
or proprietries.
And so these are quite
existential ethical
concerns, actually. They're not necessarily concerns about the
capabilities
of the software.
Often, people
who,
are anti AI and don't like the idea
of AI being used in peer review have
quite deep seated,
broader concerns than the fact that it might
not produce very good reports.
We also ask people,
do you think you would be able to
detect
an
AI authored,
peer review reports if, you were an author
and you received one on your manuscript.
And and we'd ask them, what do you
think the hallmarks are of these reports?
And
what came up again and again and again
was
these tools just do not have the depth
of knowledge of an expert
peer reviewer.
And that was really the most common responses.
They've seen these reports that are written by
generous of AI tools, and they they're very
high level.
They
make very broad generic statements, and they're just
not useful. And it's it is quite obvious
that they've been written
by Generative AI. We see, unfortunately, we see
quite a few fully
generative AI written peer review reports, and we
find the same thing. You know, they're just
not up to scratch. They're not helpful, and,
actually,
we can spot them a mile off. That
is not the same
necessarily
as reports
that have been edited or augmented
by AI. And that's where we get into
well, there's different uses, and it's quite difficult
to police how people are using them. Because
when we ask people,
in this survey,
have you used generative AI tools, and if
so, how?
The most common response was, well, I wrote
my review. Maybe they wrote it in kind
of bullet points or in note form, and
then they put the review into an a
AI tool to improve flow and grammar.
And so that was, you know, by far
the most common,
response, and
that's a completely different use case to uploading
a manuscript into chat GPT or whatever and
asking it to to summarize to to
write a review.
Although,
that was the second most reported use of
AI tools was to,
so 46 people said that they had they
had used generative
AI to
digest or summarize an article under review.
And that,
for me, is a little bit concerning because
that is not
what
current large language models are good at.
In fact, there was a really interesting
study that was published last week by,
and colleagues at Sheffield University in Tokyo in
Finland.
And
they asked they looked specifically at ChatGPT,
and they asked ChatGPT
to kind of summarize and critique
a large number of manuscripts that had been
retracted. So they'd been retracted for serious ethical
concerns.
And on the whole, chat GPT was very
positive about these papers. It very rarely picked
up on the issues that had led to
the retractions.
Generally speaking,
it it tends
to be very positive when you put
a manuscript into one of these chatbots.
You're gonna get a pretty positive,
gentle peer review, which isn't very accurate.
So do do you think there that,
is the
is chat GBT or the large language model
trying to please,
the person
who's who's using it and,
you know, saying,
sort of couching any sort of review in
positive language,
because maybe that's more acceptable to,
to the person who's making the request.
Well, it's interesting you ask that because there
was recently,
in the news,
reports that
the new versions, particularly of ChatGPT,
were to be made less sycophantic.
So previous
versions were very sycophantic, very, you know, positive
about everything the user was saying. Maybe. Maybe
it's that.
I would imagine that in the study of
retracted,
manuscripts, though, where they asked ChatGPT to summarize
the manuscripts, I would imagine the prompts
the prompts would have been quite neutral.
Who knows?
Yeah. Well, that that's the problem, isn't it,
that we don't know?
And I I just wanted to check something
with you, Laura. So
if a reviewer were to upload,
a manuscript,
and use, let's say, chat GBT to provide
them with a summary, that's something that we
don't
want people to be doing. Is that is
that correct?
So that summary
being submitted
as a peer review,
no. That's not. And, actually, as I say,
our policy also, doesn't prohibit the use of
these tools
to to edit reports either.
Perhaps it's worth talking about
people's feelings
when we ask them to put their author
hats on,
and we ask them how would you feel
if your manuscript
was reviewed in full or in part
by a large language model? And the responses
were quite telling.
So,
57%
of people said they would be unhappy
if a reviewer used generative AI to write
an entire pair of your report on a
manuscript that they had coauthored.
And when we ask people, okay,
so that's writing a full report, how would
you feel if,
AI was used to augment or edit
a pair of your report? And the response
there was still quite high. It was in
the forties of people saying,
roughly 40% of people saying they wouldn't be
comfortable with that.
So, no,
we definitely don't want fully AI authored peer
review reports. There is a real quality issue,
and there are lots of ethical issues.
And we know that
a majority of our authors
wouldn't be happy with that.
There's a whole other practical
concern here as well, which is
these
tools
are currently
free, and they're available to anyone with an
Internet connection.
If an author wanted an AI authored peer
review report,
they can do that themselves, right, in seconds.
That's not what
they submit their manuscripts
to journals for. That's not what they expect
from peer review.
And,
again,
reading into the free text comments,
we have these two words which describe the
process, peer review.
And I think often people focus on the
review, like this is about
assessing scientific rigor.
But the peer part is really important to
people, and that came across loud and clear
when people responded to to our questions because
they really do feel strongly about
this manuscript, this research that they've worked really
hard on being assessed by
someone who's an expert, a real person, perhaps
someone that they know and have met at
conferences,
but certainly someone with the the depth of
knowledge
and passion about the field to give
a good and useful review.
And, Laura, I wanted to ask you about,
well, I suppose the survey in general, but,
you know, perhaps more about the written comments
where I suppose you you're probably
you're probably,
learning about people's passions
about AI.
Do do you think that,
these two different views,
you know, this polarization,
is it going to be difficult
to reconcile
as,
as we go forward? Because,
well, AI is it's here to stay. It's
a useful tool.
People are using it more and more every
day and learning how to use it effectively.
Is it gonna be difficult to reconcile the
views of the sort of never AI people
with the, well, I can use AI responsibly
when I do a peer review people. Is
that gonna be tough?
So
I actually think the opposite.
I actually think that there's a lot of
noise at the moment, and there's been a
lot of rapid change,
and that will lead to divisions
and differences of opinion.
But as much as I sound like I've
been quite hard on these tools and and
quite negative about them, I am an optimist.
And I do believe in the power of
communities
to kind of reach a consensus and reach
some equilibrium.
The way that these tools have been presented
from the top down has often been quite
fatalistic.
So,
the line seems to have been in a
lot of cases.
These tools are here now, and if you
don't get with the program, if you don't
use them, you're gonna be left behind.
And a lot of people have got on
board with that, and they're happy with that.
They think, well, you you know, it's true.
You can't put the genie back in the
bottle. Let's just start using them. But there's,
you know, there's a backlash against that view
as well with communities and people saying, well,
no. We do have a choice, and we
we
can think in a more nuanced way about
the ethics
and how we want these tools to be
used. And I think, fundamentally,
throughout human history,
the way that tools are adopted and used
and whether they become widespread
is not something that's dictated from the top
down.
It's something that's dictated from the bottom up
through communities
who
get together, test out tools,
and decide whether these tools are gonna help
them achieve their goals.
And I think that that is what's going
to happen
with a little bit more time
in the physical science research community. You know?
Ultimately, the community is going to decide,
and and we, as publishers,
need to follow their lead.
In the short term, when we are seeing
a divergence of views, what we need to
be careful to do is make sure we
bring everyone along with us.
And that includes, you know, different groups. We've
discussed how there's a generational divide. Okay. How
do we do this? How do we have
policies
that are inclusive for early career researchers and
senior researchers?
And
the ability to opt in and out will
be important,
but the key thing will be transparency.
So,
real transparency
around
how these AI models are being used,
and how we as publishers are using AI
in our processes as well.
I see. Yeah. Because, I mean, it's, you
know, it's not just us, is it? It's
not just the scientific community that's grappling
with AI and sort of an overabundance
of
information,
some of it very poor. This is, you
know, this is the issue, isn't it, that's
facing society
at the moment?
So, you know, really, we're not alone
in this.
But I I on the other hand, I
may I suppose that, you know, we could
lead the way
as scientific
publishers providing
a
a a a way of using AI in
a responsible way to to to process information
and and make the world a better place.
Absolutely.
I think
what the future might hold is
tools that are data safe, that are ethical,
that protect research integrity, and that are kind
of embedded within our systems. So there's, you
know, complete transparency
and safety in the way that they are
used.
And as I
say,
allowing,
researchers to opt in or opt out of
their use. So there will still be people
with strong views, and they should be able
to to bypass this if they want to.
So exciting times as usual,
I suppose, in the publishing industry. Thanks, Laura.
Thanks so much for, coming on the Physics
World podcast,
Physics World weekly podcast,
That was Laura Fietham Walker, who is reviewer
engagement manager at IOP publishing.
The reviewer survey is called AI and peer
review 2025,
and it can be found on the IOP
publishing website.
I'll put a link to the report in
the podcast notes.
I'm afraid that's all the time we have
for this week's podcast.
Thanks to Laura for a fascinating
discussion, and a special thanks to our producer,
Fred Isles.
We'll be back again next week. See you
then.