Hello, and welcome to the Physics World Weekly
Podcast. I'm Hamish Johnston.
Physics World is brought to you by Institute
of Physics Publishing,
which produces more than 90
Scholarly Journals.
I OPP
is one of more than 35
organizations
around the world that's celebrating
Peer Review Week on September
23rd
27th.
In this podcast, we explore how the Publisher
is raising awareness of the importance of constructive
and respectful peer review feedback,
and how innovations
can help to create a positive
peer review culture.
That's coming up after this message.
Progress in Physics 2024
is a proud supporter of this week's podcast.
The 2 day event takes place on the
8th 9th October
at the Institute of Physics in London.
It will take a deep dive into the
exciting world of condensed matter physics.
And you can secure your place now to
avoid disappointment.
Just visit
i0p.events
air.comforward/pip2024.
Peer review is an essential part of the
modern scientific process.
In a nutshell,
a piece of research is described in a
paper, which is sent to several independent
experts for review,
As well as providing an important check on
what gets published in scientific journals, peer review
can also be a constructive process.
Reviewers will often ask for revisions before publication,
with the authors and the wider scientific community
benefiting from their expertise.
But like any human endeavor,
the process doesn't always go smoothly.
Sometimes
authors receive
very unhelpful
or outright
rude feedback
about their work.
Such reviewer misconduct
is never acceptable,
and it could dampen the enthusiasm
of early career physicists,
turning talented people away
from careers in science.
IOP publishing, which brings you physics world,
publishes over 100 scholarly
journals
and engages with thousands of authors and reviewers.
I'm joined by Laura Fethom Walker, who is
IOPP's
reviewer engagement
manager,
to talk about unhelpful reviews
and how authors and reviewers
can better engage with each other. Hi, Laura.
Welcome to the podcast.
Hi, Hamish. Thank you so much for having
me on the podcast to talk about this
topic.
So as as part of your work, Laura,
you've asked physicists
for examples of poor reviews.
Can you share a few examples that, they
got back to you with? Sure. I can
definitely give you some, examples, some of which
are quite shocking. But for context, the reason
that we we did this and we went
out and asked some senior members of the
community
about their experiences
receiving
unprofessional peer review comments
is because I think it's something that isn't
very widely talked about,
and it is an issue.
So where I've spent the last 4 years
or so running workshops with members of the
physical sciences community, early career researchers, all the
way up to very senior professors.
And often when we have discussions
that they talk about these
really unprofessional,
rude,
peer review comments that they received where they
felt personally attacked.
And it's quite clear that this is something
that really sticks with people. And often there
was quite a sense of grievance or confusion
or upset.
And what I noticed was when you talk
to really senior people who've been in the
field,
for decades
and who've had really successful careers,
some of these comments they received early in
their career, they still remember them, you know,
like it's yesterday. So it obviously have a
real impact.
And so what we've done for peer review
week this year, 2024,
is we've created a short video where we
asked for really senior,
physicists
to talk to us about the negative,
and unprofessional and rude comments that they received
early in their career. And then we, we
contrast that with their amazing achievements,
since then.
And and the reason we wanted to do
this is to kind of,
let earlier career researchers know that when they
receive,
unprofessional comments or ad hominem attacks in the
peer review reports on their papers, it's not
necessarily a reflect a reflection
on them as a researcher.
And they shouldn't let, let it get them
downhearted.
And it shouldn't affect their confidence.
So that's hopefully the goal.
We spoke to 4 senior researchers. One of
them is professor Aklesh Lactakia, who's an incredibly,
senior and accomplished,
physical scientist. And the comment that he,
highlighted
in the video,
which he received early in his career is,
this is rubbish.
Obviously, the author or authors had no electromagnetic
training and no physical intuition.
So somebody said that that was written down
on
a on a piece of paper or or
in a
some sort of review
document,
and someone thought that that was
appropriate and constructive.
Absolutely. I I don't know whether they thought
it was constructive, but it definitely
wasn't constructive. And and I think, you know,
there's
there's subtleties here and there's nuance. Right? You
you can have
a negative review. You can even have a
little bit a harsh review where the reviewer
is
strongly critiquing the science, right? They're looking at
that paper and they're saying, I really don't
think this is up to par.
I'm I'm not sure about the methodology and
things like that. When we talk about unprofessional
and rude reviews, they are always ad hominem
attacks. They are talking about the authors themselves
and not the research. And so in this
comment that, Aklesh highlighted, you know, to say
the authors have no training or intuition, that's
downright rude
and unprofessional.
Unfortunately,
it sounds like a lot of researchers have
experienced this at some point in their career.
Editors,
you know, our editors follow cope guidelines, which
is to say that
when we see unprofessional or rude comments in
a report,
we do send the review back to the
reviewer. And we say, please, can you amend
this?
Please, can you tone it down? And if
the reviewer says no, then we might make
some changes ourselves because we really don't want
these to get to the authors. However,
as you said, this is a human process.
Things slip through the net, and it's quite
clear that people are receiving these kind of
comments.
Doctor Cathy Foley, who's the chief chief scientist
of Australia, who's, again, like an incredibly senior
physicist, is also in the video,
holding up a card with the negative comments
that she received early in her career. It's
across the board,
regardless of career level. And we just felt
that this is something that really needs to
be highlighted and talked about a bit more.
I see. Okay. And I should point out
that Kathy Foley has been on the Physics
World Weekly podcast.
So, yeah.
Definitely a very esteemed
researcher.
So, Laura, not all papers are immediately acceptable
for publication, and some might have significant flaws.
What is the best way to identify
problems
in a paper in a polite
and professional manner? I mean, obviously, avoiding the
word rubbish
would be a good start. But,
what can you can you give some guidelines
maybe for how how reviewers should respond to
work that, okay, is substandard
and and the author has to be told.
Sure. I mean,
peer reviewing is a is a skill. And
and to some extent, it's more of an
art than a science. I don't think that's
widely appreciated.
It's something that people get better at throughout
their careers and with practice.
And so it's understandable if people don't get
it right all the time. We do offer
really comprehensive
peer review training here at IOP Publishing, which
is tailored for the physical sciences.
And one of the things that we talk
about in that training is focus on the
science, not the authors.
Assess your own biases. So try and be
aware of your own biases.
That might be biases against,
particular methodologies, for example. That's quite a common
one. If you receive an invitation to review
a paper and you know you are just
biased against that particular methodology,
it's probably verging on a conflict of interest.
And maybe you should, you know, say no
because you're not able to offer an objective
review.
The other tip that we give people is
that it's good practice to say at least
one positive thing in your review.
So you might,
really have problems with the paper. You might
not think it should be published for various
reasons, and you might think it's unsalvageable. You
might think that the reviewers can't revise it
in a way that would make it publishable.
And that's okay. You know, that's part of
the job of a reviewer.
But try and find one
positive
thing to say. And I think,
fundamentally,
remember
that there are humans,
human beings on the other end of this
review who are going to sit down and
read it, and who have worked really hard
to create a research paper,
and probably really care about furthering, the field.
So those are the main tips we'd give
people.
There are tips about how to structure reviews
and things like that. But fundamentally, to not
send a rude and unprofessional review, that's what
people should be keeping in mind.
And science is is by its very nature,
and it's an international thing.
You know, IOPP has authors from all over
the world. So, you know, I suppose culture
can come into this as well. I mean,
even in the, you know, I suppose the
microcosm of of Northwest
Europe.
A reviewer from the Netherlands,
might be much more blunt than a British
reviewer.
And, a Brit might, you know, they might
try to inject a bit of humor into
their review, which, you know, could fall flat
with, I don't know, an American.
So, is it possible to have guidelines to
mitigate these cultural differences,
on what is and isn't
polite discourse? I mean, you know, could you
would you tell a Brit? Well, you know,
maybe you shouldn't be you shouldn't try to
make it funny, or, you know, no sarcasm,
please, or no irony, please.
Is it possible to deal with those,
those cultural differences? And indeed, language. Because
a lot of, you know, some reviewers, English
will be their first language. Others will be
very good,
with English. But others might,
you know, maybe struggle a bit if it's
not their first language. So, you know, how,
are are those issues important when it comes
to to reviews?
It's a really interesting point and I think
it's something that hasn't been studied. It'd be
really difficult to study these kind of cultural
differences in in peer review. But it almost
certainly does happen. It's the job of the
editor really to pick these things out. And
editors are certainly trained on exactly that.
So I think the issue of,
for example, British people,
using allegories or certain kind of terminologies or
indeed trying to make jokes,
I've seen that in cases. And and the
editor should really intervene and say, you might
wanna be a little bit more clear in
exactly what you mean here. It's
it's not possible, I think, to give clear
guidelines to reviewers because, you know, it's just
not gonna be feasible to do that. I
think,
fundamentally,
when it comes to unprofessional reviews,
I think fundamentally,
when it comes to unprofessional reviews,
you can say to everyone around the world,
regardless of their culture or their first language,
if you are attacking
the authors,
or the country of the authors, the institution
of the authors, anything like that, then it's
unacceptable.
Fundamentally,
what we're really looking out for and what
we're really against is ad hominem attacks in
peer review reports.
And that's regardless of the reviewers'
geography,
culture, or anything else.
So if the reviewer is in any way
attacking the authors, their their institution,
their location, or anything like that,
that's a problem.
And if they're critiquing the science to an
extent, the tone matters a little bit less.
One thing I will say is
it's really interesting looking at the reports of,
what researchers say they have received because some
of the con some of the reported comments
are really shocking. So there was an article
published in peer j
back in 2019
by Nissa Silbiger and Amber Stubler, based in
California. And to my mind, it's the best
analysis of
this, that's been done. And they basically went
to,
a geographically diverse group of STEM researchers,
and they asked them to self report what
kind of comments have you received on your
peer review reports.
And,
they really are shocking, and I'd recommend anyone
interested to look up that paper.
Because it's things like, I didn't even bother
reading the paper because the authors are from
country x and there's no point because I
know the quality is low.
Or the author is a woman, so she
should be in the kitchen. I mean, really,
really shocking things where I think that any
any right minded person would read that and
say, that's got nothing to do with
culture or or tone. You know, that really
is just unacceptable.
So so this does happen. And what was
really interesting about that study is they didn't
just look at the comments that people received,
but they looked at the impact that it
had on people.
And what they found was,
men, women, non binary people,
people of different different ethnic minority backgrounds were
equally likely to report receiving these unprofessional and
rude rude comments.
And the rate at which everyone reported this
was quite high. I'd say quite shockingly high.
However,
the impact of those comments
was different depending on people's backgrounds. And so
women and non binary people
who were already under represented in STEM were
more likely to report that these comments had
a negative effect on them,
really knocked their confidence. And sometimes they said,
you know, it affected my career trajectory.
And the same with people
ethnic minority backgrounds.
And so this is why fundamentally,
you know, we know that this is a
problem that's not being talked about or tackled
enough. Mhmm. And I suppose that's why it's
really important as a publisher.
We don't want to
enable this. We don't want this to be
part of our process. Mhmm. So yeah. You
know, it's fantastic that you and your colleagues
here at IOPP
are tackling it. And,
listeners, we I'll I'll put
a link to that paper in the notes,
of the podcast so you can you can
take a look at that for yourself. I
mean, do you have a feeling, Laura, about
why these
personal attacks
enter,
reviews? Is it is it well,
maybe laziness isn't the right word. But, you
know, are are people just not paying attention
to what they're doing, or maybe they find
it easier
to attack the person rather than
attack the science?
Or are there
are there ulterior motives such as personal animosity
that come into play as well? Or are
people just
not thinking for the most part?
I think it's probably all of the above.
There's definitely cases where there's personal animosity.
I've seen cases where that where that's clearly
what's happening.
We know that it's so competitive nowadays to
be a researcher and a scientist.
And, you know, those kinds of conflicts of
interest do exist.
What we say to people is, if you
have a personal animosity towards some authors or
if you are essentially competing with those authors
directly to try and get some results out,
you have a conflict of interest. And if
you have a conflict
of interest, you shouldn't accept
a review invitation.
So there's that. There's the online disinhibition
effect.
There's the online disinhibition effect, which has been
studied really broadly on things like social media
sites.
However, peer review is quite an interesting case
study because,
reviewers aren't actually anonymous.
The editors know who they are. And so
there's this moderating
factor in the relationship here.
You know, the editors can see exactly
who's made these comments.
But still, it that doesn't seem to stop
some people
saying these things.
Generally speaking, you know, editors will
remove these before they get to the authors,
but that causes
real inefficiencies in the peer review process. You
know, it really slows things down.
So I think it's a lack of awareness.
I think
a lot of researchers
really, really care about their field. And I
genuinely think people get quite worked up
about papers that they received to peer review.
You know, they really care, and sometimes that
comes across in quite a negative way.
And again, you know, peer reviewing is a
skill. People need to learn
how to deliver
constructive criticism
in a way that is constructive,
that has impact for the right reasons,
and doesn't,
you know, and and doesn't attack the authors.
So it's a skill that people are constantly
learning.
Okay. And I I thought we could talk
a bit more actually about anonymity
and open processes.
The
there is a process called double anonymous peer
review,
whereby the the peer reviewer
doesn't know who's written the article, where they're
from,
you know, anything about them.
Maybe they can guess, but, you know, they
don't. It's not explicitly
told to them.
It cannot help in terms of
getting people to
to,
not to go for ad hominem
arguments if they if they don't really know
who the who the authors are.
Does that help?
Well, anecdotally, it does. But we don't have
any official figures on this. Because again, it's
something that's really difficult to track in a
in a sort of objective way. Anecdotally
speaking to the editorial teams here, it does
help.
We certainly know the statistics that we have
on our double anonymous policy show that it
reduces bias. So we do know that it
reduces bias in the publishing process.
For example,
the last year were years worth of data
that we have,
show that women who choose to anonymize their
manuscripts,
those manuscripts are 10% more likely to be
accepted.
And that is due to, you know, reviewers
being less biased.
Similarly,
authors outside of Western Europe, when they anonymize
their manuscripts, and importantly, this
includes authors in North America.
When they anonymize their manuscripts, those manuscripts are
also more likely to be accepted.
So we know that it reduces bias in
this sort of objective way.
Does it reduce the number of rude peer
review comments?
Probably.
Interestingly, you say maybe they can guess because
this is,
this is an issue that we've looked at.
We actually ask people. We have a survey
that we give out to our peer reviewers.
And one of the questions we ask them
is,
did you review a double anonymous paper?
If so, are you,
confident that you know which research group or
which person wrote this? And if so,
person wrote this? And if so, do you
want to guess? And so we've recently done
an analysis of those guesses because we've got
a few years worth of data now.
And people are nowhere near as
good at guessing as they think they are.
So this was some of the,
this was one of the things we
thought about when we implemented our double anonymous
policy
is, is it possible to ever properly anonymize
a paper?
Because often, authors are building on
historical work. They have to cite themselves.
And also in small fields, you know, everyone
knows each other. So everyone might say, oh,
that's so and so because there's only 2
groups working on this, and it's not us.
So that was one of our concerns,
as well as preprint servers.
Actually, we found that a lot of,
reviewers don't try and look the paper up
on preprint servers. We find people are generally
quite honest.
And secondarily,
we've seen a lot of cases where someone
is absolutely sure they know who the author
of the paper is. And then when we
go and check, they're off by an entire
continent, you know. So,
so that's quite heartening.
So we know it's reducing bias, and we
know that people can't necessarily figure out who
wrote the paper.
And Laura, what about transparent
peer review
where the reviewer report is published openly
for everyone to see? I could imagine,
you know, there could be some serious shaming
going on of a rude
inappropriate
reviewer
when when when that happens.
I mean, does that
force people to behave for lack of a
better way of expressing it.
It absolutely does.
So spoken to people again, the editorial team
internally, anecdotally, everyone agrees that they just haven't
seen,
sort of rude and unprofessional comments in transparent
peer review reports.
However, what I will say is the transparent
peer review
process sort of kicks in at the acceptance
stage, really.
So,
the flip side of that is
if a reviewer
made very unprofessional or harsh comments, they will
probably turn around and say, actually, I don't
want you to publish my report. So reviewers
choose whether or not to publish their reports.
They can opt in or out. We don't,
make it compulsory on any of our journals.
So there is an element of self selection
there.
Generally speaking, the reviews of transparent peer review
papers are
polite and constructive and positive. And I imagine
the reviewers are particularly proud of their work
and and are happy for it to be
published.
It's also noteworthy that reviewers can choose whether
or not to sign their reports.
So they can have their reports published in
the transparent peer review, but not have their
name next to them.
So it's all very complicated, but generally speaking,
we think that transparent peer review does nudge
people towards submitting more polite and and, constructive
reports. Yeah. I
see.
And it was it I mean, I think
it am I right in thinking that transparent
peer review is sort of a new thing?
And, you know,
transparent peer review is sort of a new
thing? And
if so,
was it difficult to get the community on
board? Because, you know, I could imagine
if you're doing a peer review and and
it is negative and it's a perfectly acceptable
negative
review, you I don't know. You might not
want to be associated
with it or,
you know, as as the sayer of bad
things.
I mean, what what is it difficult to
get people on board with it? Or
it is I'd say it can be controversial
in certain ways. There are definitely pros and
cons.
We found that uptake is actually pretty good.
Is it new in the grand scheme of
peer review of 200 years of peer review?
Yes. But other fields and and other publishers
and other journals,
have been doing this for maybe 10, 15
years on certain journals.
When
when we adopted transparent peer review, we found
the opt in rates for authors
are about 50%.
So on the journals when transparent peer review
is available, 50% of authors say, sure, you
publish the reports. Let's go for transparency. Opt
in rates are lower for reviewers. For the
reports to be published, all 3 people, the
author, and usually 2 reviewers need to say
yes. So so you can see that there's
in general, there's
quite a small percentage of cases where all
3 of those parties say yes to publishing
their reports.
The concern
with
fully open and transparent peer review where the
reviewers' names are published
is
people worry
about, particularly if they're junior researchers, they worry
about criticizing someone more senior in their field
and the potential for retaliation there.
And I suppose that is a legitimate concern.
But again, it's completely up to the reviewers
whether they publish their reports. And importantly, they
don't have to publish the names next to
their reports. So Laura, you you've spoken of
of a few things that IOPP is doing
to improve,
the quality of reviews and to make them
kinder.
Is there anything else that that you and
your colleagues are up to?
Yeah. We've been trying to kind of innovate
this peer review process for a few years
now.
The peer review process has essentially been pretty
much the same for over 200 years,
and the scientific landscape and research landscape has
changed massively in that time. So we need
to do things,
that make it easier for for our reviewers
and for our authors. One of the things
we've done is we've implemented a cut, we've
implemented a co review policy.
So this is when 2
reviewers,
usually
a junior person, perhaps even as, you know,
a senior PhD student or something like that,
can review alongside,
much more senior seasoned peer reviewers. So a
professor professor or their supervisor.
Anecdotally,
again, with with co review reports where 2
people have created this report together
and they're kind of answerable to each other
in the writing of that report, we've noticed
that the quality is higher and we see
far fewer of these unprofessional and rude comments.
So that has really, you know, helped to
improve the quality of the peer reviews we're
receiving. And it's also a training opportunity for
those early career researchers
to build this skill and this reviewing muscle.
That's really interesting, Laura, because, you know, the
reality is that science is a very collaborative
thing. Isn't it?
And it really, it just seems natural
for for 2 people to to collaborate on
a peer review. Why not? Mhmm. So yeah.
I mean, that that sounds like a really
a really interesting
idea.
And we know the reason we implemented this,
is because we know that it's it happens.
You know, it's always happened.
When you talk to early career researchers,
they'll often,
express
concern or upset that they've done so many
peer review reports for their supervisor, and yet
their supervisor is,
getting the air quotes sort of credit, whatever
that might be, the recognition.
And so we know it's happening behind the
scenes and it's frustrating people. We wanted to
bring it out in the open. And the
way that our coreview system works, everyone gets
recognition. Everyone gets credit. If they get a
trusted reviewer certification,
they both get the trusted reviewer certification.
In fact, we've had cases where 3 people
have collaborated.
And
like you say, it's a natural thing. It
makes complete sense.
And what we get when
junior researchers collaborate with senior researchers
is the junior researchers tend to have more
time. They're really keen. They're kind of,
you know, excited to get involved in the
peer review process. And the more senior researcher,
you know, really, really knows that stuff around
the science. So you get these excellent reports.
And Laura, I I also understand that you're
offering peer reviewers
feedback
on their reviews. How how's that going? Yeah.
This is an interesting one. We actually rate
all of our reviews that we receive on
a scale of 1 to 5,
with 5 representing kind of an outstanding review.
This is something that we've been doing for
years as an editorial team,
as kind of a signifier of who the
who the great reviewers are and to try
and give them recognition.
And last year, we decided to be much
more open and transparent with our reviewing community
about this. So now
all of our reviewers, when they submit their
report,
there's a question on the report form which
says, do you want to get feedback on
your review? And essentially, if they opt in
and only if they opt in, that's the
important part,
they will be told
what their review was, how it was evaluated
by the editor essentially.
And that also includes feedback on
how the rating system works and what we
expect a 5 out of 5 review to
look like.
Generally speaking, the feedback we've got from the
community about that has been very positive. I
think particularly early career researchers appreciate the opportunity
to hear back about their reviews.
Because so often, they spend a lot of
time and effort crafting this review.
They hit send,
and it just disappears into the ether. And
they don't get to find out, you know,
how much did I influence the editorial decision?
What did the editor think of my comments
and my thoughts? So we're now giving them
that opportunity.
And, again, it's hopefully, you know, improving the
overall quality of peer review and also making
our reviewers feel more valued.
Well, that that's really interesting. I mean, would
you would you incorporate, if appropriate,
feedback from authors? I mean, if an author
said,
wow, you know, that review was really good.
It, you know,
it it improved our paper. It even gave
us an idea for future research. You know,
can you thank the reviewer?
Would that would would could could you incorporate
that as well, into that feedback?
Yeah. That does happen. You know, it does
occasionally happen where an author is really grateful
to a reviewer.
I think there have been a few isolated
cases where the reviewers actually been added in
the acknowledgments of a paper because they,
you know, that their comments
really, really had an impact on the manuscript.
It's quite rare. I think if we were
to create a system where,
authors were giving feedback to reviewers, it would
be much more common
that the feedback would be negative,
unfortunately. And and, you know, it's it's our
job as editors and publishers to
moderate that relationship.
And
we think that we are much more objective
in the way that we evaluate review reports
potentially than
authors would be.
I'm not sure
how appreciative
reviewers would be about receiving
feedback
directly from authors, particularly if it was negative.
Mhmm.
Yeah. Okay. Well, that's great. Thanks so much
for coming on the podcast, Laura. And, and
happy peer review week as well. Thank you.
Thanks so much for having me.
You can find out more about Peer Review
Week at peerreviewweek.wordpress.com.
And you can browse all of IOPP's
journals
at iopscience.iop.org.
You may have noticed that this podcast has
come out on a Tuesday,
2 days earlier than usual.
That's because on Thursday,
26th
September, we're presenting the Future of Particle Physics.
This is a Physics World live event produced
in partnership
with the journal Reports on Progress in Physics.
The live panel discussion will feature
Tara Shears, of the University of Liverpool.
Phil Burrows, at the University of Oxford.
And Talika Bowes, at the University of Wisconsin,
Madison.
They'll explore what the future holds for high
energy physics, and where the next particle collider
should be built.
You can register now for this free event
on the Physics World website.
Just click on the Physics World Live tab
at the top right of the home
page. I'm afraid that's all the time we
have for this week's podcast.
Thanks to Laura Fietham Walker for joining me
today. And, a special thanks to our producer,
Fred Iles.
We'll be back again next week.
Progress in Physics 2024
is a proud supporter
of this week's podcast.
The 2 day event takes place on the
8th 9th October
at the Institute of Physics in London.
It will take a deep dive into the
exciting world of condensed matter physics,
And you can secure your place now to
avoid disappointment.
Just visit
i0p.eventsair.comforward/pip2024.