The role of peer review in ethics and research integrity
Earlier this year, I attended the Committee on Publication Ethics (COPE) and Russell Group Research Integrity Forum joint event, where Ferdousi Chowdhury from Southampton University challenged us to think about why there are so many stages of peer review, and, specifically, what purpose these stages serve in relation to the integrity of research? Could we lighten the load by combining different stages of the process or passing review information to subsequent stages? What do peer reviewers look for, ethics- and integrity-wise, at each stage? Whilst it’s important that acknowledge that the primary purpose of peer review, at whatever stage, has not historically been to assess the ethics and integrity of the research under review, some stages do include an explicit ethics review aspect.
We brainstormed some thoughts as a group, and (spoiler alert) the answer to these questions was, perhaps unsurprisingly, “It depends, but probably not, because everything is complex and varied”. In the spirit of Peer Review Week 2018, and in the light of the UK Parliament’s Select Committee Report on Research Integrity, I wanted to revisit this discussion.
Firstly, when does ‘peer review’ (in its many forms) happen, and what is it trying to achieve (if anything) in relation to research integrity at each stage? Secondly, what does this mean publication ethics and peer review?
To the best of my recollection, here are the broad stages of ‘peer review’ in the research process that we brainstormed at the workshop:
- Concept stage. This is likely to be quite informal – perhaps a chat with a supervisor or research group about a new idea or proposal. Here, there’s an opportunity for colleagues to assess and critique your approach. Feedback from colleagues here is likely to focus on the academic originality, feasibility, or grant success likelihood, rather than include ethics or integrity angles.
- Formal application (internal review stage(s)). Many departments and institutions have internal review hoops a proposal has to go through before it gets within sniffing distance of any funding. No longer being an academic, I’m unclear the extent to which ethics review features here. I’m going to hazard a guess and suggest this “varies”, likely by discipline.
- Formal application (external or money stage). At this point, many funders (in the UK at least) require some form of good practice statement or agreement (at the minimum) to accompany a research proposal. Some have designated ethics review committees and provide detailed guidance on what will be considered in the ethics review process. Others include basic requirements like submitting a data management plan and competing interest statements, but do not include a specific ethics peer review stage, nor even a requirement for reviewers to assess integrity and ethics in the proposal assessment process. So at this stage, peer review probably considers the major concerns in research ethics: working with people and animals, and data protection.
- Funding or project starts: This might involve new researchers or students coming in to the project, or a researcher starting at a new institution; each of which come with their own expectations and review protocols regarding research integrity. Many institutions/departments have dedicated ethical review teams. Is this ‘peer’ review? Often it is, as research integrity panels at institutions are often populated by academic staff. Here, peer review of the ethics may introduce new requirements or adaptations to methodology. One of the findings of the workshop was that this varies wildly by institution: some (e.g. Edinburgh and Cranfield) appear to cast the net wide and encourage everyone to go through some level of ethics review or approval. Others encourage their researchers to follow the Concordat principles, but primarily apply formal integrity and ethics steps to quite narrow and specific types of research (e.g. involving human participants or personal data).
- Publication stage: It’s notable that there’s likely a huge time gap here! Not much review of research integrity or ethics generally happens between original ethics approval at an institution, and the writing up of findings. In the interim, things are likely to have changed somewhat. Theoretically, researchers are meant to report significant changes in the direction of their research to funders and/or their institution. But what counts as ‘significant’? And what happens if a change is reported? Is the work re-reviewed? Either way, the findings are likely to be submitted for publication. And again, there’s likely to be varying baselines for what is expected of researchers in terms of prior ethics review or approval, and even more varied approaches to what is expected of editors and peer reviewers in assessing the research and publication integrity of the content. Let’s not even talk about the gulf between books and journals here!
This brings me back to the topic of this blog post – ethics and peer review in relating to publication. What should the role of peer review be in flagging ethical concerns, and does this actually happen?
As usual, COPE to the rescue with handy resources! In their Ethical Guidelines for Peer Reviewers they suggest that yes, reviewers should flag concerns about the conduct of the research or the publication. However, are peer reviewers really equipped with sufficient knowledge to judge all but the most obvious cases of fraud or misconduct? How are peer reviewers (who, perhaps wrongly, are often taking on responsibility for uncovering omissions, mistakes, falsifications, fabrications, copying, stealing) meant to pick up on things they can’t see or access? Particularly as research becomes more interdisciplinary, is it fair to ask a volunteer peer reviewer to be well-versed in the often nuanced and detailed ethics requirements of certain disciplines? Perhaps this should be the responsibility of the editors, who again, are sometimes volunteers, juggling editorial responsibilities with their own academic careers.
So now comes the fun bit. What should we do? For what it’s worth, here are some ideas to throw into the discussion during Peer Review Week:
- Pre-registration of methods – as part of the open research movement, many advocate for the use of pre-registration. Could some elements of ethics or research integrity statements be included at this stage?
- Openness generally – when research and the dissemination of those findings aren’t open, they’re much, much harder to scrutinize.
- It’s not just science! Research ethics and research integrity are often synonymous with the realms of medical or human-related research. Why shouldn’t archival research or space physics be subject to the same methodological scrutiny and expectations of research conduct early on?
- We as publishers could invest more in tools and services designed to automate or at least facilitate manuscript checking and validation.
- Authorship/contributor agreements – should these be agreed upfront (e.g. at the start of a project) and submitted alongside the article (alongside any agreed amendments)?
- Peer reviewer guidance and training – some journals provide checklists for reviewers. Even if these only serve as an aide memoire to a reviewer to consider the integrity dimensions of the research, this may help flag potential concerns earlier.
- Responsibility and accountability – much was made in the UK Parliament’s report on Research Integrity and subsequent discussion of how a lack of accountability and transparency is compounding research integrity challenges in the UK. Researchers who do commit research fraud or are found guilty of misconduct are frequently free to move to another institution with a clean slate, and to publish as usual. Whilst, as COPE points out, researchers should not be banned from publishing following research integrity or publishing ethics issues, should there be mechanisms (beyond hearsay and speculation) to at least flag content originating from a serious or serial offenders?
- Research integrity as a core part of the culture and assessment of research – everyone’s busy; funding is short; people make mistakes; life is complicated. But we need trusted communication of research so that it can be built on, learned from, and fairly critiqued. Maybe it’s time to change attitudes and perceptions towards ethics and research integrity, and treat it in the same way we do reach, impact or significance.
With the increased scrutiny and publicity around challenges to research integrity, it’s tempting for us all to jump back into our silos and think about what we’re going to do in a fairly narrow way. But this really is an issue that carries through the whole research journey (publication and peer review included). There are already multiple calls for more collaboration (the COPE/Russell Group Forum being one such example, the Parliament Select Committee Report on Research Integrity being another). Count us in!